Signal Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

Summary

Audits that use an LLM as a judge to check for bias typically rely on a strong difference-in-differences design that contrasts conditions within the same item. This paper carries out an actual pre-registered LLM-judge audit and shows that this design can cause problems in certain situations. Specifically, when the design is applied to a censored rating scale, it can manufacture an effect that does not actually exist. This suggests that some existing bias-audit findings produced this way could be spurious. The material available here is limited to the abstract-level claim, and the specific sample size and quantitative results can only be confirmed by reading the full paper.