# Which document-parsing benchmarks can you trust?
> We checked 15 document-parsing benchmarks: we ran seven open models through the 13 with public data, had three judge models count their real errors, and checked which benchmarks rank parsers the way their errors do.
- Canonical: https://www.datalab.to/blog/which-parsing-benchmarks-to-trust
- Published: 2026-11-04
- Authors: Vik Paruchuri, Paul Scemama
If you're choosing a document parser, the most useful thing you can do is run a benchmark yourself on the parsers you're choosing between. Published leaderboards mix runs from different dates, settings and people. Running it yourself leaves one question: does the benchmark's data, ground truth and scorer give you a fair answer?
We checked that for 15 benchmarks. We ran seven open models through the 13 with public data, had three judge models count the real reading errors in every output, and checked whether each benchmark ranks the models in the same order as their errors. LogicsDocBench and XDocParse keep their test data private, so we score them only on the checks that don't need a run.
We built OmniParseBench and we sell a parser, so keep that in mind as you read. We publish the scores, the per-benchmark judge results, the error probes and the rubric, so you can check our numbers (see Data).
## The short answer
fig. 1 — Score per benchmark, by part.
How to read the chart:
- Each benchmark gets up to 100 points in four parts, minus penalties: reads correctly (50), fair to every parser (25), you can run it (15) and separates parsers (10);
- The biggest single item is agreement, worth 25 points: the share of the 21 pairs of our seven models that the benchmark puts in the same order as the judges' error counts, where 1.00 is every pair and 0.50 is a coin flip;
- About half the benchmarks rank parsers about right: for 7 of the 13 we ran, the low end of agreement's 90% interval, the range it stays in when we resample the pages, is at least 0.70; and
- Ranks 2 to 6 are within 5 points of each other (74.3 to 79.2), and their agreement intervals overlap OmniParseBench's, so read the top six as close.
## How we tested
Every public benchmark went through the same steps. A benchmark's headline is the main number it reports. Pick a step to see it.
Limits:
- Seven models give only 21 pairs, so one model can move agreement a lot; and
- The judges are models, not people, though they agree with each other closely.
We fixed the rubric before computing anything, then made nine changes, most after seeing results. Two of them:
- Judging 134 to 200 pages per benchmark instead of 60 moved six benchmarks' agreement by 0.14 to 0.24, so we use all of those pages and score agreement from the low end of its interval; and
- Surya OCR 2's formulas first scored near zero, which led to the math delimiter fix in step 1.
The [rubric](/blog/benchmark-trust/RUBRIC.md) lists every change and why, and how closely the judges agree with each other.
## Results per benchmark
In rank order. Each tab gives the benchmark's rank, score and agreement with the judges, what it is, what we found and what to do if you run it.
## How much benchmarks agree with each other
fig. 2 — The same seven models on thirteen benchmarks.
Two benchmarks agree only a little more often than a coin flip. Across the 78 pairs of benchmarks, they put a given pair of models in the same order about 59% of the time, and a coin flip gets 50%. Five different models come first on at least one benchmark.
Three things drive the disagreement:
- The benchmarks cover different documents, so some of it is real;
- Format and convention: olmOCR-bench wants page headers removed and OmniParseBench wants them kept, and on ParseBench only models that turn charts into tables score on charts; and
- Each builder's model tends to do better on its own benchmark. With one or two benchmarks per builder, we can't tell how much of that is conventions and how much is tuning.
## Five checks before you trust a number
Each takes a few minutes. In parentheses: what we saw in our runs.
1. Score the ground truth as if it were a parser's output. It should score near 100 (OmniDocBench v1.5 scores every formula 0 without scikit-image);
2. Make 5% of the digits wrong. The score should clearly drop (it drops 0.1 to 0.4 out of 100 on the edit-distance benchmarks);
3. Write the same correct table in two formats, such as markdown and HTML, or compact and pretty-printed HTML. The score should move less than 1 out of 100 (pretty-printed HTML drops DP-Bench's table score by 38);
4. Leave out one page. The score should drop, or the scorer should warn you (the score doesn't drop on 7 of the 13, and only some of them warn); and
5. Read 20 failures against the page image. Most should be real errors (on OCRFlux-bench, a harmless format change moves the score more than real errors do).
## Checking ourselves
We sell a parser, so we could be tempted to score a benchmark low when our products do badly on it. We tested for that:
- We took the eight benchmarks, besides ours, whose leaderboards list our products;
- For each, we compared where our best product places on its leaderboard with the benchmark's score here; and
- If we were biased, benchmarks where we place badly would score low here.
The result goes the other way: benchmarks where we place badly tend to score higher here (a rank correlation of +0.66, where +1 means the two orders match exactly). With only eight benchmarks this could still be luck: there is an 8% chance of a link this strong by chance (p = 0.08). With OmniParseBench included, the link is much weaker (+0.15).
Most of the score comes from runs, error probes and judge counts, which don't depend on who leads a leaderboard. We can't rule out that our priorities favor benchmarks that look like ours.
## Data
- [The rubric](/blog/benchmark-trust/RUBRIC.md): every rule, formula and change;
- [The scores](/blog/benchmark-trust/scores.csv): each benchmark's points for each check;
- [The runs](/blog/benchmark-trust/runs.csv): each model's score and rank on each benchmark, and on olmOCR-bench with headers left out;
- [Agreement](/blog/benchmark-trust/agreement.csv): each benchmark's pairs in order and 90% interval;
- [Agreement by model](/blog/benchmark-trust/agreement_by_model.csv): each model's mean page score and the judges' mean errors per page on each benchmark, the two sides agreement compares;
- [Ground truth checks](/blog/benchmark-trust/ground_truth.csv): items, facts checked and facts wrong on each benchmark (one to three judges checked each item; we use the middle of three counts or the mean of two, so some counts end in .5);
- [Error and format probes](/blog/benchmark-trust/probes.csv): how each benchmark's score moves when we add an error or change the format; and
- [Ratings](/blog/benchmark-trust/ratings.csv): the four "you can run it" scores for each benchmark.
