Signal Study proposes distributional audit framework for generative AI risk in transportation

Summary

A new study introduces a Distributional Sociotechnical Audit, or DSA, to gauge the risks generative AI creates as it enters transportation through traveler advisories, synthetic crash-record generation, and policy decision support. The authors contend that existing governance frameworks lack transport-specific statistical tools capable of measuring harms that fall unevenly across population groups. Their audit merges three strands of evidence into one risk index: an analysis of 5,760 persona-controlled queries spanning four LLM families and twelve demographic attributes, statistical tests of three synthetic crash-record generators, and a Bayesian model fitted to a large Pew survey. The results show congestion-pricing advice producing the widest gap across simulated personas, while one crash-record generator failed every statistical validity test applied to it. The paper also finds that categorical approval tiers used in current governance proposals are highly unstable, with roughly 75 percent of assignments flipping under small changes to model weights. On that basis, the authors argue that continuous, sensitivity-tested risk indices give regulators a more defensible foundation than categorical tiers for governing generative AI in transport.