Signal MonitrLLM: A Community-Centered Evaluation Infrastructure for Large Language Models

Summary

Victor Ojewale, Ro Encarnación, Suresh Venkatasubramanian, and Danaé Metaxa published a paper on arXiv (cs.CY) on August 3, 2026, introducing MonitrLLM, an open-source infrastructure for evaluating large language models that links full conversation transcripts to user-reported task intent and outcomes. The authors argue that existing evaluation tools leave a critical gap: benchmark suites test controlled tasks, large-scale conversation corpora capture naturalistic use without feedback, and in-interface feedback mechanisms record satisfaction without knowing the task's purpose, but no infrastructure routinely connects all three. To demonstrate the approach, they ran a two-week feasibility pilot with 26 college students using ChatGPT, collecting 206 evaluation reports paired with full conversation transcripts. Despite participants reporting high average satisfaction of 4.19 out of 5 with their LLM interactions, the pilot found a substantial 23.1% failure rate on participants' actual goal tasks. The study also found that multi-turn conversations were reported as failing at 2.5 times the rate of single-turn exchanges, reframing extended interaction as a signal of difficulty rather than engagement. The authors argue for infrastructure that combines direct user feedback with observational data for more robust LLM evaluation.