IT Home, October 30: Users of languages with fewer speakers may need to pay higher fees for AI services compared to users of English and other major languages.
On the 28th local time, according to foreign media Rest of World, a study released last month by the Alliance for AI Language Equity found that AI models may consume vastly different numbers of tokens when processing different languages. Tokens are the basic units AI models use to process data such as text, created by tokenizers that split text, images, or audio. Researchers refer to the additional cost incurred by different languages as an implicit "tokenizer tax," with languages that have less online content being particularly affected.
Research has found that even among the 20 most commonly used languages on the internet, token consumption varies significantly, with the highest being 1.5 to 5 times the lowest. When the scope is expanded to 200 languages, this gap widens further. In one AI model, token consumption across different languages differed by as much as 17 times.
A research report indicates that a startup in Bangalore may incur higher costs each time it inputs instructions into and receives responses from AI compared to its competitors in Austin. The issue caused by the "tokenizer tax" extends beyond increased costs: the more tokens consumed, the slower the AI's response and the less context it can effectively process.
Regions with limited computing resources and network conditions are particularly affected. Inequality in tokenization also restricts people’s opportunities to participate in the AI economy.
Pratyush Chaudhuri, co-founder of Activate AI, India’s first venture fund focused on AI investments, noted that when using non-English languages, the same context capacity can accommodate fewer documents and conversation logs, and AI-generated responses require more effort.
Choudhury explained using teaching as an example: "For a teacher to effectively instruct students, they must remember what mistakes the students previously made and what explanations they have already received. If, when using a certain language, these historical records quickly fill up the context window, the quality of teaching will ultimately be affected."
Choudhury believes that language processing efficiency should be a fundamental requirement for AI products, just like accuracy, speed, and reliability. “While it may not be realistic to demand complete equality for all languages, the persistent and significant gaps must be reasonably explained and addressed with improvement plans. I hope all major AI labs will publish evaluation results for each language, detailing the cost of completing tasks of equal difficulty in different languages, and commit to improving support for the most underserved languages with measurable progress.”
IT Home learned from the report that one way to address this issue is to improve the quality of the underlying data.
However, the AI Language Equity Research Alliance found that the number of speakers of a language is not the only factor determining token consumption; the script used is equally important.
For example, Asturian has few speakers and is not included in the Common Crawl dataset, but its use of the Latin alphabet provides an advantage. Other Romance languages, such as Italian, which are represented in the data, also benefit Asturian.
Maya Research CEO Dimant Reddy noted that many existing tokenizers are still primarily optimized for languages with abundant data resources, such as English, and the fundamental issue lies in the lack of high-quality digital data.
Reddy emphasized that improving the training data alone is not enough: "Token prices are falling rapidly, and the direct impact of fee disparities should gradually diminish. Better multilingual and synthetic data can also help improve model quality. However, to truly close the tokenization gap between different languages, AI labs still need to specifically enhance the design of next-generation tokenizers to better handle these languages and scripts."