For the last year, we’ve relied heavily on the olmOCR benchmark. It’s a good benchmark, and well designed. But it’s now saturated - most gains are from more elaborate postprocessing, not model quality improvements.
Meanwhile, ParseBench from LlamaIndex has started to become more widely used. We’ve spent extensive time evaluating it. It mostly measures whether you format your output the way it expects (i.e. if you align with how LlamaIndex formats output) - we got a 650M parameter model near the top of the leaderboard with a few RL steps, without the underlying model quality improving.
Pulse, another document processing company, also got a 22 point boost on one of the benchmark’s categories merely by updating their prompt and configuration to align with LlamaIndex conventions.
We built a new benchmark, OmniParseBench, and released it last week. From now on, it’s what we’ll use to report our releases. This post explains why we needed it, and why we think it’s fair.
A year on olmOCR
We’ve used olmOCR-bench, from AllenAI, since we launched Chandra, and for every release after it (Chandra 1.5 and 2, and Surya 2).
We liked it for three reasons:
- Unit tests, not edit distance. Each test is a yes/no question about one page: is this sentence present, does this paragraph come before that one, is this cell under that header. A failing test points to a specific mistake you can go look at.
- It’s hard to game. Benchmarks built on edit distance against one reference transcription reward matching the reference’s formatting. Unit tests mostly don’t care how you format, only whether you got the content right.
- Clear signal. For a long time, olmOCR scores lined up with our vibe checks. When a model felt better, it scored better.
olmOCR is saturated
Last November, we wrote about saturating the olmOCR benchmark. We had Gemini re-check every failing test, and found most failures were formatting nitpicks, not model mistakes. With that correction, Chandra went from 85.5% to 93.9%. About 3-4% of the tests have wrong ground truth or very opinionated rules, so the real ceiling is around 96-97%.
That means there’s very little headroom left. Most remaining gains come from matching the benchmark’s formatting more closely, not from reading better.
Saturation also changes what the benchmark rewards. When models are this close to the ceiling, the remaining points come from normalizing output into the exact form the tests expect. So model releases, like the recent Infinity Parser and LightOnOCR releases, now report olmOCR scores after postprocessing written for the benchmark. That’s a reasonable response to not having other avenues to improve. But it means the score no longer correlates with model quality.
The effect is 1-2 points per model. In LightOn’s own postprocessing table, Infinity-Parser2-Pro scores 85.9 raw and 87.6 with its evaluation postprocess, and LightOnOCR-3-4B scores 85.0 raw and 86.3 with its normalization (86.0 in our reproduction). A point is larger than most of the gaps between models on the leaderboard.
And the postprocessors keep growing. Chandra OCR 2’s postprocessor is under 100 lines, and it converts our own output conventions, like <sup> tags, into the benchmark’s. LightOn’s is about 700 lines: it renders table math as text, strips dot leaders, shortens form blanks, and repairs inline math. Infinity-Parser2’s is about 700 lines too, and it applies different rewrites targeted specifically to each olmOCR category.
The incentive is clear when we apply the same normalization to our own model. Raw, Chandra OCR 2 and LightOnOCR-3-4B score the same within noise: 85.1 and 85.0. With LightOn’s normalization on top of our own, Chandra OCR 2 goes to 86.2. We could keep climbing olmOCR by writing a bigger postprocessor. We’d rather improve the model than spend our time gaming the benchmark, though.
The other problem is coverage. olmOCR is mostly English, mostly arXiv papers and Internet Archive scans. It has no forms. Its table tests only check a cell’s immediate neighbors, which simple tables pass easily. There’s almost no non-Latin script. These are exactly the documents our customers struggle with, and the benchmark can’t see them.
ParseBench
ParseBench has become a benchmark people use instead - it launched with a blog post, a public dataset and an arXiv paper, and other vendors now cite it in their launches. LlamaIndex built the benchmark, and its own product, LlamaParse, ranks first and second.
The problem with ParseBench is that it doesn’t measure OCR quality, it measures how well your output aligns with the format ParseBench expects (which usually means, how well do you align with how LlamaParse outputs things). This is not useful if you’re trying to evaluate model quality.
Here are some reasons why:
A prompt is worth 22 points
Pulse ranked 7th on the benchmark before a September update to its configuration, and 3rd after it. The update turned on a refinement step and added prompts fit to the benchmark. In September, Pulse changed those prompts to spell out the formatting the benchmark checks:
“Preserve chart captions, title hierarchy, table structure, and semantic formatting. Carry inline styling into the markdown:
**bold**,*italic*,<u>underline</u>,~~strikethrough~~,<sup>superscript</sup>,<sub>subscript</sub>,<mark>highlight</mark>, and heading levels (#, ##, ###) that match the visual hierarchy.”
Another of its prompts ends with “never add emphasis that is not there” - which is exactly the false-positive penalty in the scoring code. The commit message says it plainly: the change makes “the configuration request what the Semantic Formatting dimension measures”, and in Pulse’s own re-run, the Semantic Formatting benchmark category went from 51.39 to 73.97. That’s 22.6 points from a prompt and configuration change.
We saw the same effect without any prompt. Turning on documented output options in our API moved our Semantic Formatting score from 46.27 to 77.35 in our run of their harness (40.79 as published). The text was identical, and the formatting isn’t better, per se, it just matches the conventions ParseBench expects.
A 650M model near the top, with 70 RL steps
If ParseBench mostly measures format, a small model should be able to get to the top. So we tried it.
We took Surya, our 650M parameter open OCR model, and ran 70 steps of RL. We did not train on any ParseBench documents.
After 70 steps and a few inference fixes, Surya went from 23rd to 3rd of 148 (5th without the fixes) on the four non-chart dimensions - ahead of every frontier model and every open model on the board.
| Tables | Content | Sem. Format | Visual Ground. | Mean | Rank | |
|---|---|---|---|---|---|---|
| LlamaParse Agentic Plus | 93.37 | 92.25 | 87.12 | 84.09 | 89.21 | 1 |
| Surya, with RL and inference fixes | 84.14 | 90.00 | 86.14 | 72.31 | 83.15 | 3 |
| Anthropic Opus 5.5, effort high | 94.25 | 91.72 | 77.04 | 65.33 | 82.09 | |
| Google Gemini 3.8 Flash, thinking high | 89.12 | 89.68 | 76.36 | 72.91 | 82.02 | |
| Surya, after 70 RL steps | 83.04 | 89.38 | 85.24 | 70.33 | 82.00 | 5 |
| Surya, before RL | 82.10 | 89.00 | 74.34 | 68.11 | 78.39 | 23 |
Surya is a very strong OCR model, but other benchmarks and vibe checks indicate that it’s not actually better than Gemini 3.8 flash and Opus 5.5 at OCR.
This means that the benchmark rewards formatting, not actual model quality improvement.
We’re releasing the checkpoint, with the converter and the exact settings, so you can check this yourself: datalab-to/surya-parsebench-rl.
Charts measure estimation, not parsing
The Charts dimension asks a parser to turn every chart into a table, then checks individual numbers in it.
- 73% of the chart rules (3,569 of 4,864) are estimates. The value isn’t printed on the chart, so the parser has to read it off a bar or a line. The ground truth for those is itself a VLM’s estimate.
- Estimates are scored as if they were exact. Most allow 5-10% relative error, and 182 fall back to the 1% default.
- Labels have to match the ground truth’s naming. A parser that names a series differently, or orients the table the other way, fails, even if it has the right number.
- A parser that keeps the chart as an image, or transcribes the text on it, scores zero.
The leaderboard shows the result. The median system scores 37 on Charts, and the best frontier model 70.89. The top scores all go to products with a chart-to-table pipeline: LlamaParse Agentic Plus (94.18), Pulse Ultra 2 (89.70) and LlamaParse Agentic (88.68).
Charts sorts parsers by whether they have that pipeline, not by how well they read the page. Again, we’re not judging model quality, we’re judging alignment to the benchmark.
Visual Grounding follows LlamaParse
The code that generates the layout ground truth says it works “from normalized parse output”, and maps labels with map_llamaparse_raw_label_to_canonical.
We tried to reproduce the boxes with an open layout model. In our test, an open layout model, PP-DocLayoutV2, matched only 68% of text boxes and 39% of section titles at IoU 0.8. No public tool we tried reproduces them.
The leaderboard shows the effect. All three LlamaParse tiers score within half a point of each other on Visual Grounding (83.77 to 84.25), across a 15x price range. The best of the other 144 systems scores 80.61, and the best frontier model 72.91.
Compare that with the other dimensions. On every one of them, LlamaParse’s cheapest tier trails the best other system, by 2 to 12 points. Visual Grounding is the only dimension where it beats everyone:
| Dimension | LlamaParse Cost Effective minus best other system |
|---|---|
| Tables | −10.06 |
| Content | −1.94 |
| Semantic Formatting | −12.20 |
| Charts | −11.79 |
| Visual Grounding | +3.16 |
We don’t know exactly which system drew the boxes, but it seems likely it is related closely to how LlamaIndex draws boxes. A cheap tier that loses everywhere else and wins only here is what you’d expect if the ground truth follows its conventions.
There’s more - scoring bugs that zero out whole categories for pretty-printed HTML, rules no parser can satisfy, and documents that are never scored. But the four above are the core problem: a ParseBench score mostly tells you how closely a parser matches ParseBench’s conventions.
What OmniParseBench does differently
We wanted to keep what made olmOCR good, and fix what made it saturate.
It keeps the unit test design. Every test is still a yes/no question about one page: this text is present, this comes before that, this cell sits under these headings, this line is inside a table block. OmniParseBench has 16,288 tests on 2,937 pages, from 2,343 source documents.
It doesn’t care about your formatting. HTML or markdown, pretty-printed or not, $…$ or <math>: every output goes through the same converter and the same normalization, applied to both sides of every comparison. Equations are compared by what KaTeX renders them to, not how the LaTeX is written.
Every test says where its answer came from. Most tests come from sources that know the answer: the publisher’s JATS XML for tables, forms we filled in ourselves, PDF text layers, olmOCR’s own reviewed tests. The rest come from a panel of independent models, with disagreements decided by a person. Each test records which, so you can filter by how much you trust it.
It covers the hard cases. Forms (blank, filled and scanned), handwriting, degraded scans, tiny text on engineering drawings, rotated pages, 92 languages, and tables with real structure - multi-level headers, section rows and row groups, checked against the publisher’s XML.
It has headroom. We removed 18,264 tests that every system passed and that a plain tool with no model (pdftotext, PyMuPDF, Tesseract) also passed - they can’t tell parsers apart. The best system scores 92.9 overall, and the hard slices are much lower. On non-Latin scripts, the best system passes 88.6% of tests and the median system 81.5%. On degraded scans, the best is 85.6% and the median 78.6%. On handwriting, the best is 81.4% and the median 72.2%. We’ll continue to add more hard edge cases so it doesn’t fully saturate.
Our launch post goes deep on how tests are tagged and how to slice the results.
Making it fair
We built a benchmark that we compete on, so the bar for fairness has to be higher, not lower. Here’s what we did:
- No vendor gets a custom prompt. Every parsing API runs at its own best settings, using only its documented options. General-purpose models (GPT, Gemini, Claude) all get the same short prompts, published in the repo. Nobody gets instructions written against the rubric - including us.
- Adapters only change format. Converting a vendor’s markdown to HTML, or putting back headers a vendor returns separately. Each change is checked to lose nothing.
- Same treatment on errors. Every vendor gets the same timeout and the same retries.
- Unsupported is not failed. If a vendor can’t produce layout blocks, those tests are left out of its score, not counted as wrong.
- Ground-truth models are marked. Gemini 3.8 Flash and Claude Sonnet 5.5 helped create or verify some tests. We still score them, but we mark them, because they may have an edge on those tests.
- The headline can’t be tilted by test counts. It’s the average of text, tables and layout, so adding more tests of one kind makes that score more precise without making it count more.
- Everything is public. The code is Apache 2.0, and every page and every test is published.
- We don’t train on it. Benchmark pages are kept out of our training data.
It will keep growing
A static benchmark eventually saturates - that’s what happened to olmOCR. So OmniParseBench is a living benchmark. When we (or a customer) find a failure mode it can’t see, we’ll add tests for it. Several suites are already in the pipeline: more handwriting, federal forms, newspapers where models drop whole columns, and characters from the wrong script leaking into text.
Each release is versioned, so scores are always compared on the same version.
How we’ll report releases
From now on, every model and API release we announce will report OmniParseBench scores. Here’s where things stand today:
| Parser | Headline | Text | Tables | Layout |
|---|---|---|---|---|
| Gemini 3.8 Flash* | 92.9 | 91.3 | 95.2 | 92.2 |
| Datalab accurate | 92.7 | 90.8 | 91.8 | 95.6 |
| Datalab balanced | 91.6 | 90.0 | 89.2 | 95.5 |
| Reducto | 90.8 | 85.4 | 91.2 | 95.8 |
| LlamaParse | 90.4 | 88.2 | 92.6 | 90.4 |
| Mistral OCR | 89.5 | 82.8 | 91.9 | 93.8 |
| Claude Sonnet 5.5* | 88.7 | 93.1 | 95.4 | 77.5 |
| GPT-5.6 Sol | 87.8 | 85.4 | 91.6 | 86.5 |
| Extend | 83.3 | 64.1 | 92.5 | 93.4 |
| Azure | 83.0 | 74.6 | 81.7 | 92.8 |
* Used in creating and verifying some tests. Full results, cost and latency are in the launch post.
We built this benchmark, and we sell a parser - you should keep that in mind, just as you should with ParseBench. The difference we’re aiming for is that you don’t have to take our word for anything. Every test has a recorded source, and the scorer is about 2,000 lines of plain Python you can read.
And as always: the best benchmark is your own documents. Run your own evals.