Signal AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

Summary

Researchers introduced AISPA (AI System Prompt Assurance), a user-centric framework for systematically auditing the system prompts that govern commercial AI product behavior. Using the framework, they reviewed 3,249 instructions drawn from system prompts in 88 commercial AI products, classifying each as either protective of users or problematic. The audit found that system prompt design varies substantially across products, with some organizations averaging more than 60 protective instructions per product while others average fewer than five. While 98.9% of products contained at least one protective instruction, only 24% covered all eight dimensions of the AISPA taxonomy. Roughly 40% of products contained at least one instruction that worked against user interests, and protective and problematic instructions frequently coexisted within the same prompt. The researchers also found that system prompts have grown steadily longer and more protective over time.