solidot新版网站常见问题，请点击这里查看。
消息
本文已被查看 126 次
研究发现非英语的 token 成本更高
Coalition for Language Equity in AI Research 上个月发表的一项研究发现，AI 模型处理部分语言提示所需的 token 量远超英语。研究人员称之为隐性 token 税，意味着小语种用户的 AI 成本更高。研究发现，即使互联网最流行的 20 种语言，token 使用量也存在差异。最昂贵语言使用的 token 数量是最便宜语言的 1.5-5 倍。当研究人员将范围扩大到 200 种语言，差异进一步扩大，一个 AI 模型的 token 成本差距甚至增加了 17 倍。研究指出：位于班加罗尔的初创公司可能需要为每次提示和完成支付附加费，而其位于奥斯汀的竞争对手则无需支付。token 税的影响远不止成本。token 数量越多，延迟就越高，有效上下文容量就越低。在计算和连接本就有限的地区，这些惩罚最为严重。token 不公平限制了 AI 经济的参与范围。
https://clear-languages.org/blog/the-tokenizer-tax
https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Click here to see the most common problems on the new version of the website.
Message
This file has been viewed 126 times
The study found that non-English token costs are higher.
A study published last month by Project for Language Equity in AI Research found that the AI Model requires far more token than English for some language tips. Researchers call it a hidden token tax, which means that the AI costs are higher for small language users. The study found differences in the use of token even in the 20 most popular languages of the Internet. The number of tokens used in the most expensive languages is 1.5-5 times the number of cheapest languages. When researchers expand the range to 200 languages, the difference widens, and the token cost gap for an AI model increases even 17 times. The study noted that the start-up company in Bangalore may need to pay a surcharge for each tip and completion, while its competitor in Austin is not. Token taxes have implications far beyond costs. The greater the number of tokens, the higher the delay, the lower the effective context capacity. These penalties are most severe in areas where computation and connectivity are limited. Token ' s unfairness limits the participation of the AI economy.
https://clear-languages.org/blog/the-tokenizer-tax
https://restofworld.org/2026/ai-tokenizer-tax-non-english-languages-cost-inequity
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.