Signal Preprint measures tokenization cost disparities across languages

Summary

A preprint proposes a reproducible benchmark called the Tokenization Equity Audit. Large language models are increasingly deployed as general-purpose educational and technical assistants. Yet the paper argues that the underlying infrastructure does not treat languages equally. It identifies tokenization as an underexamined source of disparity, noting that semantically equivalent content requires substantially different token counts across languages. This affects API cost, latency and usable context length even before a model is invoked, disadvantaging underserved language communities.