Signal Data Annotation as Measurement

Summary

A preprint argues that data annotation, a foundational input for modern AI systems, should not be narrowed to a question of inter-annotator agreement but understood as an act of measurement. The authors contend that current practice treats agreement among multiple annotators as sufficient evidence of quality, yet agreement alone cannot establish whether the annotations validly capture the underlying concept they are meant to represent. Drawing on measurement theory, they call for reframing annotation-quality assessment around validity rather than consensus. This amounts to a proposal to re-examine the fundamental reliability of AI training data. The authors aim to shift the focus of the debate over annotation quality itself.