# Benchmark trust rubric (100 points)
Question: if you run this benchmark yourself on the parsers you are choosing between, will the result be fair and useful?
The same rules apply to all 15 benchmarks, OmniParseBench (ours) included. We wrote these rules before computing any score, then changed nine things, most after seeing results. Each rule below is the one we used. The changes are listed at the end with their reasons.
## Data
- Runs: 7 open models (olmOCR-2, MinerU2.5-Pro, PaddleOCR-VL-1.6, dots.mocr, Logics-Parsing-v2, chandra-ocr-2 and Surya OCR 2) on the 13 benchmarks with public data. Each benchmark's own scorer runs at a pinned commit, on 200 sampled pages (199 on ParseBench), in a sandbox with no network. Before scoring, every model's math delimiters are rewritten to one form: `` and `\(x\)` become `$x$`, and `` and `\[x\]` become `$$x$$`. No character of the LaTeX changes.
- Judges: three judge models from three vendors, Gemini 4 Argon, Claude Opus 5.5 and GPT-5.6 Sol. Each judge sees the page image and the output, and, for ground-truth checks, the ground truth. For agreement, we use the middle of the three counts. Their counts agree closely: Fleiss kappa is 0.76 over 5,145 outputs, with counts grouped as 0, 1 to 2, 3 to 5 and 6 or more. Run again on the same outputs, each judge gives the same group 89 to 98% of the time.
## Reads correctly (50)
- Ranks parsers like their real errors (25). The judges count real reading errors in each model's output on every sampled page that all seven models scored (134 to 200 per benchmark): wrong, missing or invented words, numbers, table cells and formula parts. Format and convention differences are not errors. For each of the 21 pairs of models, the pair is in order if the model with the higher mean page score on the benchmark also has fewer mean errors per page. On olmOCR-bench, both sides use outputs with headers left out, its documented rule. Agreement is the share of pairs in order. We resample the pages 1,000 times for a 90% interval, and use its low end L. Points = 25 × (L − 0.5) / 0.4, kept between 0 and 25.
- A real error costs more than a format change (10). Five realistic error types (wrong digits, words in the wrong script, character confusions, wrong table numbers and deleted words) at 1, 2, 5 and 10% of the places each can occur, added to the ground truth, or, on the pass/fail-test benchmarks, to real models' outputs. 5 × the share of error types whose cost rises at every step, plus 5 × min(1, R / 5), where R is the median cost at 5% divided by (the median score change from harmless format changes + 1).
- The ground truth is right (10). The judges check 1,100 to 5,300 ground-truth facts per benchmark (a word, number, table cell, formula or test assertion) against the page image. One to three judges checked each item, and we use the middle of three counts or the mean of two. 10 if 1% or less are wrong, 7 if 3% or less, 4 if 6% or less, else 0.
- The headline is about reading (5). 5 × the share of the headline that is reading: text, tables, formulas or layout.
## Fair to every parser (25)
- No tests only one format can pass (10). 10 × (1 − the share of the headline that only one kind of output can pass). HTML-only tables, chart-to-table conversion, required markup and a fixed rule on page headers count. Layout and LaTeX formulas don't, because modern parsers produce both.
- Ranks hold under unstated conventions (15). We re-score the 7 models under each convention the benchmark doesn't document as required: headers and footers kept or removed, tables as HTML or markdown, table whitespace normalized or not, and math delimiters normalized or not. Points = 15 × (1 − the mean over the 7 models of how far each model's rank moves, out of 6).
## You can run it (15)
Two raters score four questions, and a third settles disagreements: test data public and complete (4), scorer runs as published (4), output requirements documented (4) and inference code or prompts included (3).
## Separates parsers (10)
- 7 points if the smallest score gap that a page resample orders right 80% of the time is 1 or less, 0 if it is 5 or more, linear between.
- 3 points if the best of our 7 models is 10 or more below the ceiling, 1.5 if 3 to 10 below, else 0.
## Penalties
- −5 if an empty page, the PDF text layer alone or the ground truth with its blocks shuffled scores 80% or more of the ground truth's own score.
- −3 for each scoring bug that changes which of two real parsers ranks higher, −1 if fixed, at most −10.
- −3 for public evidence that test pages were in a compared model's training data.
## Private benchmarks
LogicsDocBench and XDocParse have no public test data. Their computed items get 0, and their rated items are scored as usual.
## Changes after we wrote the rules
1. Before any score: the shuffled-ground-truth check shuffles blocks (paragraphs and tables kept whole), not lines, because line shuffling crashes the OmniDocBench scorer.
2. After a partial preview: "ranks hold under unstated conventions" leaves out conventions a benchmark documents as required. olmOCR-bench documents headers and footers removed and math through KaTeX; OmniParseBench documents math through KaTeX; OCRFlux-bench documents its table form.
3. After a partial preview: olmOCR-bench's baseline tests, which check that output is not empty, garbled or repeated, count as reading.
4. After seeing results: agreement compares the 21 model pairs by their means instead of every pair of models page by page. Pass/fail-test benchmarks give two models the same page score on 39 to 57% of page pairs, so the page-level number measured how finely a benchmark scores a page, not whether it ranks parsers right.
5. After seeing results: the ground-truth check counts errors per fact (a word, number, cell, formula or test assertion) instead of per page or test, because a page with hundreds of facts and a test with one are different units.
6. After seeing results: the ground-truth check samples about 1,000 facts or more on every benchmark, applies one punctuation rule to every item, and re-checks items where the judges' counts differ by more than 3 times. olmOCR-bench's agreement uses its documented header rule.
7. After seeing results: the math delimiter rewrite in Data. Surya OCR 2 writes LaTeX inside `