Signal Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

Summary

William Bolton and Philip Torr published a paper on arXiv (cs.CY) on August 4, 2026, proposing complex, adversarial, fast-moving real-world domains where expert practitioners independently generate observable outputs as a solution to the notoriously difficult problem of benchmarking AI scientists' ability to generate novel ideas. They instantiate this framework in two structurally different domains: Formula 1, where models ideate around car design concepts for the 2026 season against real pre-season innovations as ground truth, and Magic: The Gathering, where models propose decks from a recently updated card pool evaluated against 19 Pro Tour decklists. Across both domains, models produced plausible outputs but few aligned with real-world expert solutions. In F1, the best model, GPT-5.2, matched 10 of 40 real innovations across 166 proposed ideas. In MTG, the best deck from Gemini 3 Flash recovered 5 of 7 new-set cards from the third-place Pro Tour deck, and across all 108 decks, the cards models selected most often correlated with the cards most widely adopted by Pro Tour decks (Spearman correlation 0.74, p=0.0003). The authors conclude that a key capability gap for AI scientists is not idea generation but filtering, prioritization, and coherent novelty.