I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.
Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:
Language Relative tokens
--------------------------------------
English 1.00x
Portuguese 1.23x
Chinese (Simplified) 1.25x
German 1.31x
Spanish 1.32x
French 1.37x
Arabic 1.38x
Chinese (Traditional) 1.42x
Korean 1.47x
Swahili 1.49x
Hindi 1.57x
Japanese 1.66x
Burmese 3.16x
Amharic 5.78x
Santali 13.70x
Source: "Tokenizer Fairness in 2026", a reproduction/extension of
Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce
Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.
Yeah this site has a severe issue of some kind that I don't think I've ever seen before. NGL, I'm pretty impressed! It takes so much compute that scrolling lags, which feels very, very strange
Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
Perhaps relevant: Chinese language is not more efficient than English in vibe coding: A preliminary study on token cost and problem-solving rate https://arxiv.org/abs/2604.14210v1
This is interesting, thank you. It also reinforces the stocastic parrot theory of LLM. Maybe one mythical day when the majority of codebases around the world are written in CJK Han, vibe coding will become more efficient in Chinese
I think the demo font might not have Chinese characters at all. Most Latin fonts don't include CJK, and the system will fill in missing characters with available Chinese fonts. CJK fonts OTOH do include Latin characters.
As for tokenizers - I reckon that most of them are not optimized for CJK; they're good enough, and AI performance in CJK languages, so far, haven't been the top priority matter.
I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.
Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:
Source: "Tokenizer Fairness in 2026", a reproduction/extension of Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.
https://github.com/partyfly/tokenizer-fairness-2026
I typed the first stanza of 'Jabberwocky'.
Worked about as well as expected :D
This page reliably hangs Firefox 155 at 100% CPU for me.
I did most of my testing in Firefox 156 and it's fine for me
smooth as butter on 152 in windows 10, as long as we're all reporting
Yeah this site has a severe issue of some kind that I don't think I've ever seen before. NGL, I'm pretty impressed! It takes so much compute that scrolling lags, which feels very, very strange
Haha this did help me generate empathy for the assistant. Neat idea.
The kerning is absolutely awful in Safari. Looks fine in Chrome though...
I wonder how this would look in Chinese Mandarin.
Completely the same as regular text, except punctuations are centered instead of staying at the bottom of the line. CJK Han likely encodes tokens to character one on one. Which brings an interesting question, are Chinese characters more efficient for NLP? In the sense that semantic meaning of a word is not chopped up into partial "tokens".
Perhaps relevant: Chinese language is not more efficient than English in vibe coding: A preliminary study on token cost and problem-solving rate https://arxiv.org/abs/2604.14210v1
This is interesting, thank you. It also reinforces the stocastic parrot theory of LLM. Maybe one mythical day when the majority of codebases around the world are written in CJK Han, vibe coding will become more efficient in Chinese
I think the demo font might not have Chinese characters at all. Most Latin fonts don't include CJK, and the system will fill in missing characters with available Chinese fonts. CJK fonts OTOH do include Latin characters.
As for tokenizers - I reckon that most of them are not optimized for CJK; they're good enough, and AI performance in CJK languages, so far, haven't been the top priority matter.