Signal ASSERT: A Measurement Pipeline for GenAI Audits

Summary

The paper observes that audits of generative AI systems commonly summarize behavior as a single reported compliance rate, which researchers and stakeholders then use to compare systems, track regressions, and decide whether to approve deployment. The authors argue that this reported rate reflects not only the audited system's actual behavior but also the measurement choices made behind the scenes. As a result, when the rate changes it becomes unclear whether the underlying system changed or the measurement methodology did. To address this, the paper proposes ASSERT, a measurement pipeline designed to separate these two sources of variation. The pipeline is intended to help those interpreting audit results distinguish the influence of system behavior from that of measurement design choices.