Signal Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling

Summary

The design of robust risk models to assess societal risks posed by advanced AI systems is an emerging area in AI governance, and while many regulatory proposals increasingly require systemic risk assessment, what state-of-the-art risk modeling should look like in practice remains an open question in the absence of rigorous quantitative methods. The researchers identify the key methodological and institutional challenges that currently limit the adoption of risk modeling and review five research traditions that inform the problem: probabilistic risk assessment, catastrophic AI risk analysis, cybersecurity risk quantification, Bayesian causal inference, and threshold-based governance. They compare two leading proposals, scenario-based risk estimation and Bayesian network-based threshold setting. Drawing on a workshop with 22 experts and subsequent analysis, they identify a structured agenda of open questions concerning model structure, scope, evidence integration, validation, and governance. The authors close by arguing that progress will depend on integrating quantitative modeling with independent evaluation, transparent and tiered disclosure, and institutions capable of maintaining and updating risk models over time.