Signal Blind Turing Test Evaluates LLMs on Italian Legal Professional Exams

Summary

A study by Germana Bertoli, Ilaria Amelia Caggiano, Francesca Lagioia, Riccardo Rovatti and colleagues reports the results of a blind Turing Test evaluating leading commercial LLMs on Italy's three legal professional exams, for lawyers, judges and notaries. The researchers asked leading LLMs to produce complete written exam papers. These papers were made indistinguishable from those written by humans. The papers were then evaluated anonymously by expert examiners using the same grading criteria applied to real examinations. This design let the researchers assess not just whether the LLMs could pass but how their performance compared directly with qualified human candidates under authentic professional testing conditions. The same method was applied across all three exams.