Signal Preprint tests recovering test-item parameters from text embeddings instead of human calibration

Summary

A preprint tests an alternative to the traditional way of calibrating test items in multidimensional item response theory. Multidimensional IRT normally relies on item parameters such as discrimination and threshold values estimated from large samples of human responses. This study asks whether the directional loadings of those parameters can instead be recovered straight from item text using pretrained sentence embeddings, without any human responses. In other words, the key question is whether the initial human-based calibration step can be skipped. The method was tested on the public IPIP Big Five personality dataset. That dataset consists of 50 items answered by 19719 respondents.