Signal New Benchmark Reframes LLM Context Robustness as a 'Selective Trust' Problem

Summary

A preprint by Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu and colleagues notes that language models increasingly base their answers on external signals, and a single misleading one can flip a correct answer into a wrong one. The usual fix, training models to resist such signals, creates a new failure mode: a model that ignores all context looks robust on the surface yet becomes useless once the context is actually trustworthy. The authors reframe the problem as selective trust rather than blanket resistance and introduce MIST, a human-annotated benchmark built to evaluate this more nuanced capability. The work targets a core reliability challenge for AI agents that must weigh conflicting or uncertain outside information. The authors suggest this reframing could become a standard for evaluating related models going forward.