Signal Ordinary, Reasonable Chatbots: Do AI Models Track Human Legal Judgments?

Summary

As people increasingly rely on AI for guidance, scholars, lawyers, and judges have begun considering AI's role in legal decision-making, with 'silicon jurors' potentially following 'silicon sampling' from social science research into courtrooms. This study joins an emerging line of research on generative AI's ability to simulate human legal judgments, examining how LLM-powered chatbots respond to questions about legal reasonableness. The authors compare answers from human participants to those of twenty-six LLMs across twenty-five different legally relevant reasonableness judgments. Overall, chatbot responses generally track those of human participants, but compared to humans, LLMs generate more homogeneous responses and occasionally treat a variable standard as an invariant rule. Compared to humans, LLMs also tend to generate answers more favorable to the government and to corporations, and their responses align more closely with those of respondents who are white, male, older, and more educated. The authors state that more systematic research is needed to confirm or reject these initial findings.