Open-source OCR models on OmniParseBench

Two weeks ago we released OmniParseBench: 16,288 pass/fail tests on 2,937 pages, with a leaderboard of document-parsing APIs. A natural next question is about open models: if you want to run a model on your own GPUs, which one should you pick, and how far behind the APIs is it?

To answer it, we ran 17 open-weight OCR models on the same tests. Each model ran with its vendor’s documented serving setup, prompt and output parsing; the few departures are listed below. We publish every number, including the ones where our own models, chandra-ocr-2, Surya OCR 2 and Indra OCR, lose. The same numbers are on the dataset card, with every verdict behind them; only Datalab accurate’s headline (93.5, a newer run) and Indra OCR’s row (version 1.1, from its repository) differ.

The rankings

The headline is the mean of three family scores: text, tables and layout (how the benchmark works). A model that returns no layout boxes has no layout score, so it has no headline; we show its text and table scores. Parameters are those used at inference, counted from each model’s weight files: an output layer tied to the embeddings counts once, and multi-token-prediction heads, which only speed up decoding, are left out.

model	parameters	headline	text	tables	layout
chandra-ocr-2	4.5B	90.1	87.4	87.4	95.4
dots.mocr	3.0B	87.9	82.5	88.6	92.5
MinerU2.5-Pro	1.2B	87.1	77.4	92.1	91.7
Infinity-Parser2-Pro	35.1B (MoE)	87.0	83.3	84.1	93.5
Surya OCR 2	0.67B	83.1	80.6	74.3	94.3
LightOnOCR-3-4B	4.5B	82.9	83.9	70.8	94.0
LightOnOCR-3-0.8B	0.85B	81.6	81.1	72.1	91.7
GLM-OCR	1.3B	81.5	71.0	84.2	89.3
Nemotron 3.5 Parse	0.63B	80.0	72.4	77.3	90.2
Unlimited-OCR	3.3B	79.3	70.2	76.5	91.1
PaddleOCR-VL-1.6	0.96B	78.7	61.7	83.7	90.7
Infinity-Parser2-Flash	2.2B	78.4	78.4	66.5	90.4
DeepSeek-OCR-2	3.4B	75.2	64.2	70.0	91.4
Indra OCR	12.3M	71.4	62.7	59.6	92.0
HunyuanOCR	1.0B	64.6	50.5	61.9	81.4
olmOCR-2	8.3B	none	71.3	56.6	unsupported
Nanonets-OCR2-3B	3.8B	none	68.2	62.6	unsupported

The best open models are close to the API leaderboard. chandra-ocr-2 (90.1) is above Mistral OCR (89.5), Claude Sonnet 5.5 (88.7) and GPT-5.6 Sol (87.8), and 3.4 points below Datalab accurate (93.5). dots.mocr, MinerU2.5-Pro and Infinity-Parser2-Pro follow within 3.1 points. After them there is a gap of 3.9 points: the next 11 models score 64.6 to 83.1. NVIDIA’s Nemotron 3.5 Parse, released on October 17, scores 80.0 with 0.63B parameters.

The family scores show that the models fail in different places:

• Text separates the models most: from 87.4 (chandra-ocr-2) to 50.5 (HunyuanOCR). The tags below show where the text is lost.
• Tables range from 92.1 (MinerU2.5-Pro) to 56.6 (olmOCR-2). The cause is mostly table structure, not reading (see below).
• Layout is high for every model that draws boxes: 81 to 95.

Size and score are only loosely related. Surya OCR 2 leads the models below 1B parameters, and chandra-ocr-2 leads overall at 4.5B. Infinity-Parser2-Pro, the largest model at 35.1B, scores below chandra-ocr-2 and dots.mocr. At the other end, Indra OCR is more than 50 times smaller than any other model here: 12.3M parameters with its math models, 1.25M of them in its main recognizer, and it runs on a CPU. It scores 71.4, within 4 points of DeepSeek-OCR-2, in its default mode with its math models on. That mode reads a digital PDF’s own text layer where it passes its checks and OCRs the rest. Without the math models it scores 66.7, and reading every page as an image, 61.2.

What OmniParseBench tests

OmniParseBench is a set of 16,288 unit tests on 2,937 pages from 2,343 documents. Each test is a yes/no question about one page, so a failure points to one specific mistake you can go look at:

test type	checks	family
present	this text, number or equation appears	text
order	text A comes before text B	text
repeat	this line appears exactly N times	text
table_cell	this cell sits under these headings and beside these neighbors	tables
layout_kind	the block holding this line is a heading, text or table	layout

The tests cover the documents that parsers find hard: tables (6,767 tests), including multi-level headers and row groups checked against the publisher’s XML, math (3,316), multi-column pages (3,517), forms that are blank, filled and scanned (1,028), handwriting (923), tiny text (832), rotated pages (824), degraded scans (655), and 92 languages. Most answers come from sources that know them, such as the publisher’s JATS XML, forms we filled in ourselves and PDF text layers, and each test records where its answer came from.

Every output goes through the same converter and normalization, so HTML or markdown, $…$ or <math>, all score the same. We also removed tests that every system passed, that a plain tool with no model also passed, and that a judge model rated trivial, so there is headroom: the best system scores 93.5, and much less on the hard slices.

Where the open models lose text

Because every test is tagged, any slice can be scored. Here are the hardest slices, scored as the headline is (the mean of the family scores on the tagged tests), with four API systems from the launch post for reference:

model	handwriting	math	table	form	multi-column	tiny text	rotated	degraded	not English
chandra-ocr-2	66.8	96.1	90.7	74.6	90.4	89.1	82.2	85.9	74.6
dots.mocr	58.9	96.1	89.3	68.5	86.4	81.1	83.8	71.7	74.4
MinerU2.5-Pro	52.0	88.0	91.9	75.9	87.5	85.1	82.3	77.6	60.4
Infinity-Parser2-Pro	65.2	88.5	88.2	72.6	88.8	84.1	71.7	87.0	66.0
Surya OCR 2	55.2	94.7	83.7	66.0	82.0	74.7	59.9	72.6	71.4
LightOnOCR-3-4B	65.4	96.4	85.5	75.2	83.0	79.5	75.7	87.8	64.8
LightOnOCR-3-0.8B	57.1	87.1	85.2	73.0	81.6	78.4	77.1	76.8	60.2
GLM-OCR	54.1	85.8	86.4	67.0	84.2	71.7	72.7	76.8	48.2
Nemotron 3.5 Parse	51.2	92.1	83.0	67.2	82.1	72.8	61.8	72.3	61.6
Unlimited-OCR	43.9	95.4	79.6	64.0	79.3	67.0	36.6	65.7	51.8
PaddleOCR-VL-1.6	52.3	85.4	84.8	63.8	79.2	72.2	77.7	68.9	52.9
Infinity-Parser2-Flash	58.1	96.9	77.3	66.5	81.7	73.3	56.8	78.7	54.5
DeepSeek-OCR-2	43.6	91.4	75.3	56.0	77.4	62.1	36.5	60.8	52.1
HunyuanOCR	42.1	53.0	70.6	53.2	72.1	54.8	49.2	65.6	53.0
Indra OCR	41.4	59.5	75.7	53.2	72.1	69.0	69.1	55.9	59.2
Datalab accurate (API)	72.4	97.0	94.5	81.0	93.9	92.9	92.2	89.6	81.0
GPT-5.6 Sol (API)	67.6	92.4	91.6	79.2	87.4	81.1	77.9	87.0	75.2
Gemini 3.8 Flash (API)	79.4	98.3	93.8	85.0	92.3	88.5	92.5	85.4	83.8
Mistral OCR (API)	69.0	95.6	91.5	76.7	89.6	81.2	89.4	78.6	77.0

Each cell is shaded by its distance from the median of all 27 systems on the dataset card for that slice: blue above it, red below it, darker up to 15 points away. Medians: handwriting 58.9, math 92.4, table 86.4, form 73.0, multi-column 84.2, tiny text 81.1, rotated 77.7, degraded 77.1, not English 65.8.

Datalab accurate’s slices are from its October 15 run, which scored 92.9; its current headline is 93.5.

Tests per tag: handwriting 923, math 3316, table 6767, form 1028, multi-column 3517, tiny text 832, rotated 824, degraded 655, not English 1950. Not English means a test with a language tag other than English.

Handwriting and non-English pages are the hardest slices for most open models. On handwriting the best open model scores 66.8, and Gemini 3.8 Flash scores 79.4. Rotated pages split the models: most open models score 49 to 84, and Unlimited-OCR (36.6) and DeepSeek-OCR-2 (36.5) score below 40. Math is close to solved for some of them: 5 of the 14 models in the table above 95 on it (chandra-ocr-2, dots.mocr, LightOnOCR-3-4B, Unlimited-OCR, Infinity-Parser2-Flash).

Why OmniParseBench, and not olmOCR-bench or ParseBench

Open models are usually compared on olmOCR-bench, and more and more on ParseBench. Both have problems that matter for this comparison. Why we built OmniParseBench has the full detail.

olmOCR-bench is saturated

About 3-4% of olmOCR-bench’s tests have wrong ground truth or very opinionated rules, so the real ceiling is around 96-97%, and the top models are close to it. The last points now come from postprocessing written for the benchmark. In LightOn’s own table, Infinity-Parser2-Pro scores 85.9 raw and 87.6 with its evaluation postprocess, and LightOnOCR-3-4B scores 85.0 raw and 86.3 with its normalization. Those postprocessors are about 700 lines each. A point is larger than most of the gaps on the leaderboard.

Raw, three of the models in this post are within a point of each other on olmOCR-bench. On OmniParseBench they are not:

model	olmOCR-bench, raw	OmniParseBench
Infinity-Parser2-Pro	85.9	87.0
chandra-ocr-2	85.1	90.1
LightOnOCR-3-4B	85.0	82.9

olmOCR-bench: chandra-ocr-2 is our run; the other two are as LightOn reports them.

olmOCR-bench also can’t see much of what customers send. It is mostly English arXiv papers and Internet Archive scans, it has no forms, its table tests check only a cell’s immediate neighbors, and it has almost no non-Latin script.

ParseBench measures formatting

ParseBench, from LlamaIndex, mostly measures how closely a parser’s output matches the formatting it expects:

• A prompt is worth 22 points. Pulse changed its prompts to spell out the formatting the benchmark checks, and its Semantic Formatting score went from 51.39 to 73.97. Turning on documented output options in our API moved ours from 46.27 to 77.35, with identical text.
• A 650M model reaches the top with formatting alone. We ran 70 RL steps on Surya, our 650M model, without any ParseBench documents. With a few inference fixes it went from 23rd to 3rd of 148 (5th without the fixes), ahead of every frontier model. On OmniParseBench, Surya OCR 2 without that RL scores 83.1: fifth among the open models in this post, and below GPT-5.6 Sol, Claude Sonnet 5.5 and Gemini 3.8 Flash.
• Charts measure estimation. 73% of the chart rules are values that are not printed on the chart, and they are scored as if they were exact.
• Visual grounding follows LlamaParse. An open layout model matched only 68% of the text boxes at IoU 0.8. LlamaParse’s cheapest tier trails the best other system on every other dimension, by 2 to 12 points, and leads on this one.

We compete on it too

We built OmniParseBench and our models are on it, so we hold it to a higher bar: no system gets a prompt written against the tests, adapters only convert formats, every system gets the same timeouts and retries, and every page, test and adapter is public. That is also why “How we ran the models”, below, lists every setting we used for every model.

Tables: structure, not reading

In the table failures we examined closely, most were not missing text. The test finds the cell’s text, but the cell is under the wrong heading, beside the wrong neighbour, or in the wrong row group. Merged cells are a frequent cause. On a table with a two-level header, a model that writes no colspan or rowspan flattens the header, and every value below it moves to the wrong column.

LightOnOCR-3 shows the gap between reading and structure clearly. Its 4B model’s text score (83.9) is higher than dots.mocr’s (82.5), but its table score is 18 points lower (70.8 against 88.6).

Looping replies

Small OCR models sometimes repeat a phrase until they hit their token limit. The vendors handle this differently. chandra-ocr-2’s client retries a repeating reply at a higher temperature, and olmOCR-2’s pipeline retries up to 8 times. MinerU2.5-Pro, Unlimited-OCR and DeepSeek-OCR-2 use an n-gram logits processor on the server. HunyuanOCR’s client stops a repeating reply and keeps the text before it. Most clients keep a reply that is cut at the token limit, so a loop shows up as failed tests on that page. Nemotron 3.5 Parse’s client is stricter: it returns no document for a reply cut at the limit, which happened on 31 of its 2,937 pages. We kept each vendor’s own behavior and did not add any.

How we ran the models

The aim was to run each model the way its vendor documents it, and nothing else:

• Serving: the vLLM version and flags the vendor documents, on H100s (one per server; two for Infinity-Parser2-Pro).
• Prompt and sampling: the vendor’s documented prompt, temperature and token limit. Where a vendor publishes a client or pipeline (chandra-ocr, dots.mocr, LightOnOCR, MinerU’s mineru-vl-utils, GLM-OCR’s SDK, PaddleOCR-VL, DeepSeek-OCR-2’s vLLM code, HunyuanOCR’s client, NVIDIA’s nemotron-parse), the adapter calls the vendor’s own code for prompts and output parsing.
• Layout: if the model’s documented mode returns typed boxes, the adapter maps its labels onto the benchmark’s labels with a fixed table. Models without boxes score only on text and tables.
• Page input: a model whose vendor code reads PDFs gets the page file and renders it the way its vendor does, at the vendor’s resolution (listed below). A model whose vendor code takes images only (HunyuanOCR, Nanonets-OCR2, Nemotron 3.5 Parse) gets the page as a PNG with its long edge at most 2,048 px.
• No post-processing: the adapters only convert formats (markdown to HTML, a vendor’s tags to plain HTML). No answer is edited.
• Time limit: 900 seconds per page, as every API on the leaderboard has.
• Scores: the numbers are the dataset card’s, on dataset revision 961c0f89, whose pages and answers are identical to the launch revision. Every verdict is in the dataset’s results.parquet.

A few choices depart from a vendor’s documentation, and we list them so you can judge them:

• chandra-ocr-2: the package renders the PDF at 300 DPI, with form fields flattened. The chandra-ocr package’s ocr_layout prompt at temperature 0 and top_p 0.1, at most 12,384 tokens, with the package’s retry of a repeating reply (up to 6 times, temperature 0.2 higher each time). The adapter also retries a reply cut at the token limit, which the package cannot detect; a reply cut on every attempt is kept as it is.
• dots.mocr: the parser renders the PDF at 200 DPI. The repository parser’s prompt_layout_all_en prompt, temperature 0.1, at most 32,768 tokens. A reply whose JSON does not parse falls back to the parser’s cleaned text, with no layout, as the parser does.
• LightOnOCR-3: rendered with poppler at 400 DPI, lowered so the page fits 5 million pixels, as LightOn’s own benchmark does. The grounding prompt at temperature 0.1. The client sends no token limit, so we used 12,288, the limit of LightOn’s own benchmark pipeline.
• Unlimited-OCR: the pipeline renders the PDF at 300 DPI. The vLLM recipe’s prompt, n-gram processor and settings. The recipe’s example limits output to 8,192 tokens; we allowed 28,000, the model card’s 32,768-token context less room for the prompt.
• MinerU2.5-Pro: MinerU renders the PDF at 200 DPI. The mineru-vl-utils client’s two-step extraction (layout, then content per block) with its default prompts. The client’s own retries are off, because the benchmark retries transient failures the same way for every system.
• olmOCR-2: the pipeline renders the PDF at its 1,288 px default. The toolkit’s prompt and retry schedule (up to 8 attempts at rising temperature, one after another). We left out the toolkit’s last-resort fallback, which returns the PDF’s embedded text layer when no attempt succeeds, so a page with no successful attempt scores as failed.
• GLM-OCR: the SDK renders the PDF at 200 DPI. The SDK’s documented pipeline (PP-DocLayoutV3, then the model per region). Its server failed when it took many pages at once, so we ran 12 server processes with one page each, which is the SDK’s single-request path. By default the SDK drops the regions it labels header, footer, page number, footnote, aside text or reference before reading them; we changed only its config.yaml so those regions are read as text, because the benchmark scores the whole page with headers and footers kept, as it does for every system. A page with a region the SDK failed to read is sent again, where the SDK would drop the region without a warning.
• Nanonets-OCR2-3B: the model card’s document prompt at temperature 0, at most 15,000 tokens, with the card’s repetition penalty of 1.0. We did not use the card’s optional prompt for financial documents.
• PaddleOCR-VL-1.6: PaddleX renders the PDF at 144 DPI. The documented pipeline (PaddleOCRVL, version 1.6, PP-DocLayoutV3 layout, the model served by vLLM), with the layout model on the CPU. By default the pipeline leaves headers, footers, footnotes and side text out of its markdown; we kept every block, because the benchmark scores the whole page with headers and footers kept, as it does for every system.
• DeepSeek-OCR-2: rendered with PyMuPDF at 144 DPI, as the repository’s PDF script does. The repository’s own vLLM path: vLLM 0.8.5 with DeepSeek’s model code, image processor (crop mode) and no-repeat n-gram processor, the <|grounding|>Convert the document to markdown. prompt at temperature 0, at most 8,192 tokens (its context length), and the evaluation script’s post-processing. The repository runs this as a batch script; we wrapped the same engine setup in a small HTTP server so the benchmark could send it pages.
• HunyuanOCR (1.5): the vendor’s validated vLLM 0.18.1 setup with its serve.sh flags and its client’s request and streaming repetition stop; text up to the stop is kept, as the client keeps it. We used the layout_parse task, which returns the whole page with typed boxes; the default doc_parse task leaves out headers and footers and has no layout. In August the model repository’s config.json moved to a newer format that vLLM’s position-encoding code does not read, so we served the current weights with the repository’s previous config.json; the weights, tokenizer and processor files are identical between the two commits.
• Surya OCR 2: the package renders the PDF at 192 DPI. The surya-ocr package (0.22.1), with the surya_ocr command’s full-page recognition and the package’s default prompt, sampling and token limit, served on vLLM 0.20.1 with the flags the package builds for an H100, including its default multi-token prediction (MTP). We turned on the package’s option to ask again, at a rising temperature, when a full-page reply does not parse or repeats itself (up to 6 times); after that the package falls back to layout plus per-block OCR, as it does by default.
• Infinity-Parser2 (Pro and Flash): the SDK renders the PDF at 300 DPI. The infinity_parser2 SDK’s document-to-JSON prompt and its JSON parsing, at temperature 0 with at most 32,768 tokens and thinking off, as the model card sends it, served on vLLM 0.17.1 with the model card’s flags; Pro on two H100s with tensor parallelism. A reply cut at the token limit keeps its complete elements, as the SDK does. By default the SDK’s markdown leaves out headers, footers and page footnotes; we kept them, as for every system.
• Indra OCR: version 1.1 on CPU (x86, 4 vCPUs, one page at a time), in its default auto mode with --math on: it reads a digital PDF’s text layer where the layer passes the package’s checks, and OCRs every other page and region. With math off it scores 66.7, and with --mode ocr, which reads every page as an image, 61.2. Scores are from Indra OCR’s repository, on the same dataset revision and scorer.
• Nemotron 3.5 Parse: nvidia/Nemotron-3.5-Parse at commit 62cb1e4, with NVIDIA’s nemotron-parse client (3.5.0.post7) in its default single-pass mode, served on vLLM 0.26.0 with the flags in its docs/serving.md (one-token MTP, FlashAttention 3), on one H100 with 64 pages in flight. The client takes images, so it gets the 2,048 px page image, and the model’s processor resizes it to at most 1,664 x 2,048. We used the client’s default sampling (temperature 0.2, top_k 50, repetition penalty 1.05, at most 8,744 tokens), which NVIDIA’s own ParseBench pipeline also uses. The model card’s quick start uses greedy sampling instead; with it, the model scored 78.8. NVIDIA’s ParseBench headline uses a triple pass that keeps extra copies of each table for the evaluator to choose from; with it, the model scored 82.0. We report the default single pass.

We train chandra-ocr-2 and Surya OCR 2, so we have an interest in this comparison. That is why every setting is listed here, and why the adapters are published in the omni_parse_bench repository, with Nemotron 3.5 Parse’s added in PR #26, so that anyone can rerun them. Every score for every system is also in the repository’s results table. If you think we ran a model below its best documented setup, open an issue and we will rerun it.

Which model to pick

• For the best overall score: chandra-ocr-2, dots.mocr, MinerU2.5-Pro and Infinity-Parser2-Pro. They are within 3.1 points of each other.
• For tables: MinerU2.5-Pro (92.1) and dots.mocr (88.6).
• For text, handwriting and non-English pages: chandra-ocr-2 leads all three. Next are LightOnOCR-3-4B on text (83.9) and handwriting (65.4), and dots.mocr (74.4) and Surya OCR 2 (71.4) on non-English pages. The gap to the best APIs is still large here: on handwriting, chandra-ocr-2 scores 66.8 and Gemini 3.8 Flash 79.4.
• If you need layout boxes, check the layout column: every model with boxes scores 81 or higher there (the lowest is HunyuanOCR), but olmOCR-2 and Nanonets-OCR2 do not return boxes.
• If you only have CPUs: Indra OCR. On 4 cores its median page takes 357 ms with its math models on, and 212 ms with them off.

To explore any slice yourself, the benchmark guide shows how to score runs by tag, suite or test type.

If you would rather not run a model yourself, the Datalab API runs Chandra with the additional correction steps behind the “Datalab accurate” row.