On an old hymnal page, a faded line reads “For every vain and idle thought”. Chandra OCR 2.4 wrote it as “For every罪 and evil thought”. 罪 is Chinese for “sin”. The model didn’t drop in a random glyph. It wrote a word that fits the sentence, in a writing system that appears nowhere on the page.
For every and evil thought,
That was the weird part. Once we started looking, it kept happening the same way: the meaning or the sound is right, and the alphabet is wrong.
- On a Finnish bank report, LEPPÄVIRRAN ends in ران, Arabic letters that spell “ran”.
- In a Finnish newspaper ad, Tarjoamme (“we offer”) ends in the Greek letters ουμεν.
- In a Thai PDF table, the abbreviation สบอ ends in the Arabic letters بو.
We found it in our own audits. We’re always monitoring the quality of our OCR models and fixing what we find, and this is one of the strangest issues we’ve seen.
We call these script confusions. They are rare, a fraction of a percent of pages, and they are sneaky. The page reads fine at a glance, edit-distance metrics barely move, and a search for the right word won’t find it.
Here’s what it looks like, why we think it happens, how often frontier models do it, and how we fixed it.
- On the page
- สบอ.-๑
- Chandra OCR 2.4
- สبو.-๑ Arabic letters
- Chandra on the Datalab API today
- สบอ.-๑
The Thai abbreviation สบอ ends in the Arabic letters بو. The model wrote it this way every time the abbreviation appears on the page.
Why it happens
Script confusions are an OCR-specific case of what Marchisio et al. (EMNLP 2024) call word-level language confusion: a model inserts single words or phrases in a language other than the one it should be writing. Two things make the OCR case different. First, the correct text is on the page in front of the model, so a script confusion is a transcription error, not a choice of language. Second, the meaning or sound is often right while the script is wrong: the model writes the same word or sound in another alphabet. We have not seen this second pattern discussed for OCR before.
Our working theory: the model knows roughly what the word or sound is, but the script it writes it in comes from its language prior, not from the page. That fits work on the latent language of multilingual transformers, which found that Llama 2 models already have the meaning of the next token in their middle layers, often in its English form, and only move to the language of the input in the last layers. When the image is ambiguous, or the surrounding text is in another script, the prior wins.
Two measurements support this.
Where the model confuses scripts, the prior wins over the page. We asked an earlier Chandra model for its top 5 next-token probabilities at two positions where it writes the wrong script on real pages, with greedy decoding.
On an old English page that says “terribly earnest”, the word “earnest” is tiny and faded, and it is not in the top 5 at all. The model guesses from context, and its two best guesses are “cruel” in Chinese (残酷, 22%) and “cruel” in English (11%). The meaning comes from the language prior, and the script is close to a coin flip.
Some script mixing is correct. Documents switch languages, like the Arabic line on this Hebrew page. That is code-switching, and code-switching is not the problem: a correct transcription switches script exactly where the page does. The problem is a word written in the wrong script on top of it. Here, the Arabic word نوسع ends in ع, the Arabic letter ayin. The model writes ע, the Hebrew ayin: the same letter and sound in the page’s main script, at 74%. All five of its candidates are Hebrew.
At the same positions, Chandra on the Datalab API today stays in the right script. On the Hebrew page it writes ع (78%), though it misreads the letter before it. On the English page it stays in English, but it still misreads the faded word. Fixing the script does not make an unreadable word readable.
Equivalent words in different scripts sit close together inside the model. To write each token, the model scores every token in its vocabulary against its current state. Each token has a learned vector, a kind of fingerprint, and tokens with similar fingerprints get similar scores. If “cruel” and 残酷 have nearly the same fingerprint, a state that favours one also favours the other, and a small push from the language prior decides which one gets written. That is what we find: of the roughly 53,000 tokens written in Chinese characters, the closest one to “cruel” is 残酷.
The pattern holds well beyond one word. For 93% of the Chinese words we tested, the English translation is among the 10 closest of about 118,000 Latin-script tokens, and Cyrillic words often sit next to their Latin and Greek transliterations. Greek is the exception: the Latin transliteration of a typical Greek word is no closer than a random token. Greek-Latin confusions come mostly from look-alike letters instead, which we cover below.
How we measured: we used the public Chandra OCR 2 weights, which share one embedding matrix between input and output (bf16, 248,320 tokens × 2,560 dimensions), and measured closeness as cosine similarity between rows. For each pair, we ranked the equivalent token among all tokens of its script. Transliteration pairs (4,500 Cyrillic → Latin, 412 Cyrillic → Greek, 920 Greek → Latin) are built automatically from the tokenizer. Translation pairs (150 Chinese, 40 Katakana) are a hand-picked list of common words, so they likely overstate the average word.
We think this is the mechanism, but we have not proven it. Close embeddings and near ties are consistent with it. They don’t show that the embedding geometry causes the confusion, rather than both coming from the same training data.
Frontier and open-source models do it too
We measured script confusions on about 5,600 real document pages. The set is intentionally hard: we built it to surface script confusions, so it leans heavily toward non-English pages in many scripts, degraded scans, and difficult benchmark pages. On typical documents, which are mostly clean English, every model confuses scripts less often than the rates below. A detector flags letters from a script the page does not use and words that mix scripts, and we checked the flags against the page images by hand.
| Model | Pages flagged by the detector | Share of flags confirmed by hand | Estimated pages with a real script confusion |
|---|---|---|---|
| Chandra OCR 2.4 | 1.27% | 15 of 16 sampled | 1.19% |
| Chandra on the Datalab API today | 0.27% | all reviewed | 0.16% |
| Gemini 3.8 Flash | 0.83% | 17 of 23 sampled | 0.61% |
| Claude Opus 5.5 | 1.01% | 22 of 25 sampled | 0.89% |
| LightOnOCR-3 (4B) | 3.04% | 30 of 30 sampled | 3.04% |
| DeepSeek-OCR 2 | 6.08% | 30 of 30 sampled | 6.08% |
LightOnOCR-3 and DeepSeek-OCR 2 were run with their documented prompts on vLLM with greedy decoding. Their confusions include whole passages and loops in a wrong script, not only single words. Failed requests and empty outputs are left out of every rate.
Open-source OCR models show the same error, often more severely. LightOnOCR-3 transcribed a Lao archaeology handout in Thai letters: the title ການວັດແທກວັດຖຸບູຮານ came out as การวัดและวัดทุนฐาน.
Many of the script confusions are homoglyphs: a letter from another script that looks the same as the right one, like a Latin o inside a Cyrillic word. They look correct on screen, but they break search and string matching just as badly. In the hand-checked samples, homoglyphs are 20 of Opus’s 22 confirmed script confusions and 10 of Gemini’s 17. They are also more than half of the few left in Chandra today.
This is the problem that Unicode Technical Standard #39, Unicode Security Mechanisms, addresses with mixed-script detection and its list of visually confusable characters: a string that mixes look-alike letters from different scripts can pass for a different string. A homoglyph confusion puts exactly that kind of string into OCR output. That is why it matters beyond looks: it breaks search and string matching, and it is the same kind of string that spoofing relies on.
Some of the frontier model script confusions we confirmed against the page image:
| Model | Output | What the page says |
|---|---|---|
| Gemini 3.8 Flash | CAGCTTGGTCТТAAGACTCT (two Cyrillic Т) | All Latin T |
| Gemini 3.8 Flash | Датчик ОКCІМАКС (one Latin C) | All Cyrillic |
| Claude Opus 5.5 | Telugu వాழும் (last letters Tamil) | Telugu వాఴుం |
| Claude Opus 5.5 | Serbian ученикoвог (one Latin o) | All Cyrillic |
| Claude Opus 5.5 | Раеd (Cyrillic Ра, Latin ed) | All Cyrillic |
The DNA primer is the worst kind. It looks correct on screen, but two of its letters are Cyrillic, so a sequence search will not find it and a string comparison will say it is different.
Both frontier models are very good at reading these pages. The point is not that they are bad at OCR. This class of error survives in the strongest models, because almost nothing measures it.
We fixed it
We fixed it with a novel training technique that we developed. On the same intentionally hard pages, the share of pages with a script confusion drops from about 1.2% for Chandra OCR 2.4 to 0.16% for Chandra on the Datalab API today. That is about 7 times fewer, and lower than every other model we tested. For a 20-page document, the chance of at least one script confusion drops from about 21% to about 3%. All eight pages in the viewer at the top of the post now come out right; each one shows the page, the output with the wrong script, and today’s output. On the hymnal, today’s model writes “For every vain and idle thought,“. The same model is also more accurate overall than Chandra OCR 2.4.
It is not zero. Today’s model still confuses scripts on about 1 in 600 of these hard pages. Five of the nine we confirmed are look-alike letters, like a Latin letter inside a Cyrillic word, and the other four mix in letters from a different script. We will keep working on both.
Why this is hard to benchmark
Script confusions hit under 1-2% of pages, and they change a word or two. Standard metrics barely see them.
- Edit distance. On the eight example pages at the top of the post, repairing only the wrong-script letters changes the page’s character edit similarity by 0.02 to 0.65 points out of 100. A benchmark built on edit distance hardly moves.
- Benchmarks. OmniParseBench has 34 tests on the “terribly earnest” page above. None of them contains the sentence with the wrong-script word, so the error costs nothing on the benchmark.
- Scale. The documents tell a different story. These rates come from our intentionally hard set, so treat them as an upper range. At Chandra OCR 2.4’s rate, about 11,900 of every million pages have a word in the wrong script, and 21% of 20-page documents contain at least one. At today’s rate, it is about 1,600 pages per million and 3% of 20-page documents.
If you train or evaluate on OCR output, those words land in your data as token sequences your model has rarely seen. If you search it, the words are not there. A benchmark score will not tell you this is happening. You have to measure it directly and look at the pages.
What’s next
Script confusions are one of a family of errors that are rare, cheap for a metric, and expensive for the people who use the output. We will keep finding them, measuring them, and fixing them.
You can try Chandra on the Datalab API. If you find a script confusion in our output, send us the page.