Signal A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation

Summary

A preprint proposes a four-axis trustworthiness benchmark for using large language models as judges in principle-based regulation. The authors note that principle-based standards such as being fair, clear and not misleading cannot be reduced to a simple binary verdict, yet LLM-as-judge systems are increasingly substituting for human evaluation of compliance with such standards. Their position is that any such judge must be assessed on four axes: accuracy, paraphrase robustness, adversarial robustness and calibration. The paper releases a benchmark called Principle-Bench, made up of 168 test cases, to support this evaluation. The aim is to provide a systematic way to check the trustworthiness of AI systems tasked with regulatory compliance judgments.