Signal Mixed-methods study maps how LLMs are disrupting Capture the Flag cybersecurity competitions

Summary

A paper reports a mixed-methods study of how large language models are disrupting Capture the Flag (CTF) cybersecurity competitions, combining a synthesis of published benchmarks (including a recent government evaluation), case studies of live competitions across three challenge categories, structured observation of public community channels where players debate AI use, and semi-structured interviews with experienced players and organizers. The study maps the current human-machine capability boundary by category, finding that easy and intermediate challenges in cryptography, web exploitation, and binary exploitation are now reliably automated by LLMs, while narrower sub-categories continue to resist automation. It finds that community disagreement over whether AI should be permitted is downstream of an undeclared prior question about what a competition is actually for. In response, the authors propose a four-component safeguard framework combining tiered competition divisions, LLM-resistant challenge design, investigatively-used telemetry, and a draft community code of conduct, along with a decision tool that ties the choice of safeguards to a competition's declared purpose. The authors argue the implications extend beyond CTFs to any cybersecurity setting where a demonstrated result is taken as evidence of an underlying human ability.