Signal Correcting Mode Collapse in Silicon Sampling with Semantic Similarity Rating

Summary

Researchers examined 'silicon sampling,' the use of large language models to generate survey responses, and identified a persistent problem of mode collapse in which generated response distributions show unrealistically low variance. The paper argues this collapse stems from LLMs' difficulty generating numeric data directly, and that text-based responses may be better suited to the task. To address this, the authors tested a method called Semantic Similarity Rating, which solicits text-only responses from LLMs and then maps those responses to a numeric scale using text embeddings. They applied the method specifically to questions about political attitudes. The method was found to improve the fidelity of silicon sampling response distributions compared to direct numeric elicitation. The authors also note that the approach has relatively few parameters that need to be calibrated.