Study shows some languages require up to 17 times more tokens to use AI models
A new study by the AI Language Equity Research Consortium found significant differences in the number of tokens AI models need to process different languages: among the 20 languages with the highest web presence, less common languages use 1.5 to 5 times as many tokens as English to process the same prompt. After expanding the study to 200 languages, the gap reached as high as 17 times in one of the models examined. Because most LLM providers charge by token, this drives up usage costs and also brings longer response times and smaller available context windows, which researchers call a hidden tokenizer tax.
A new study by the AI Language Equity Research Consortium found significant differences in the number of tokens AI models need to process different languages: among the 20 languages with the highest web presence, less common languages use 1.5 to 5 times as many tokens as English to process the same prompt. After expanding the study to 200 languages, the gap reached as high as 17 times in one of the models examined. Because most LLM providers charge by token, this drives up usage costs and also b
Matched fieldsBody



