# Extraction benchmarks aren't fair, so we made one you can audit
> OmniExtractBench: 620 documents pulled from four different vendors' benchmarks, scored by one deterministic grader that explains every decision. Code and data are public.
- Canonical: https://www.datalab.to/blog/omni-extract-bench
- Published: 2026-09-16
- Authors: Paul Scemama, Zayd Sukhun, Vik Paruchuri
At Datalab, we've dedicated ourselves to a hard, unglamorous problem: getting AI to read documents correctly, without any hallucinations or mistakes. Extraction across thousand-page tables and messy forms is one of the hardest tasks in document processing. A model that hallucinates a single wrong figure in a contract or a filing can have real business impact.
Our view is that an extraction benchmark should do two things:
- Help customers choose the right vendor; and
- Give engineers a way to diagnose what's actually going wrong in a given model.
We've been paying attention to all of the recent benchmark launches, and benchmarks like ExtractBench and LongExtractBench are useful starting points, but they have common failure modes:
- The documents and scoring can be biased to the specific vendor who made the benchmark;
- The prediction harness is hard to audit, so you can't tell whether a low score means the model failed or the harness did;
- The scoring is convoluted or hard to interpret, so "why did this document score low?" has no real answer; and
- The documents only cover a small subset of the universe of potential extraction cases. Some benchmarks (like LongExtractBench) only have dense tables. Others only have clean documents, with no scans. This doesn't represent the real world nor the real challenges.
We wanted a fair way to evaluate extraction options. So we decided to take four existing benchmarks from different vendors (LlamaIndex, Extend, micro1 and Datalab), clean them up, standardize scoring and make them as fair as possible.
Our benchmark, **OmniExtractBench**, includes:
- **620 documents across four existing benchmarks from several vendors.** This eliminates bias;
- **A scorer that's fully deterministic and explains every decision through a `Verdict`** — an atomic, auditable unit that tells you exactly whether a value was matched, misread, missed, invented, or fabricated, instead of collapsing everything into one opaque score. This makes it auditable;
- **Clean, consistent and fair rules for the tricky internal pieces**, like how to score `null` values vs blanks. This makes the scoring transparent and easier to understand; and
- **Documents that typically break extraction**: huge repeated-value tables, dense scalar schemas, research papers, credit agreements, resumes and filings. This covers a wide range of real-life challenging use cases across several verticals.
Here's how several major extraction options actually did, scored the same way, on the same corpus of documents:
fig. 1 — Accuracy, precision and recall across the 620-document corpus.
| System | Mode | Accuracy | Precision | Recall | Coverage |
| ---------------- | ---------------------- | --------- | --------- | --------- | -------- |
| datalab-accurate | accurate | **93.85** | **95.32** | **95.11** | 620 |
| datalab | balanced | 93.48 | 95.30 | 94.79 | 620 |
| reducto | deep_extract v2 | 93.47 | 94.91 | 95.02 | 620 |
| claude | opus 5 | 90.96 | 95.17 | 92.60 | 575 |
| extend | extraction_performance | 90.17 | 91.67 | 94.66 | 620 |
| gemini | 3.7-flash | 86.79 | 94.48 | 88.81 | 526 |
| llamaextract | agentic_plus | 84.93 | 86.57 | 93.13 | 616 |
| gpt | 5.6-sol | 83.85 | 95.11 | 84.99 | 615 |
| mistral | ocr 4.1 | 76.78 | 85.93 | 79.27 | 574 |
| azure-cu | gpt-4.1-mini | 61.08 | 80.32 | 64.07 | 569 |
Each mean is calculated over how many documents the provider could actually process — the coverage column. Short coverage is a limit of the provider: the document either exhausted its output limit, or its schema was rejected as too large.
We will continue to improve the tooling, auditability and fairness of this benchmark. We are keen to hear your thoughts (especially if you disagree). The scoring code is [on GitHub](https://github.com/datalab-to/omni_extract_bench) and the data is [on Hugging Face](https://huggingface.co/datasets/datalab-to/omni_extract_bench).
## Data
620 documents across 4 sources:
| Source | # Documents | Coverage |
| ------------------------------------------------------------------------ | ----------- | -------------------------------------- |
| [extractbench](https://github.com/run-llama/ExtractBench) | 329 | Forms, filings, decks. |
| internal | 202 | Dense scalar schemas, small documents. |
| [micro1](https://www.micro1.ai/benchmark/long-extraction) | 47 | Very large tables. |
| [longarray](https://huggingface.co/datasets/Extend-AI/LongArray-Extract) | 42 | Large tables with repeated scalars. |
Here is a breakdown of the documents:
fig. 2 — Breakdown of document types by source.fig. 3 — Breakdown of documents per page count range.
## Scorer
### Challenges of scoring
The challenge with extraction scoring is with ambiguous addresses of values. An address is a path to a scalar value that you walk along a dictionary to get to that value. For example, the single address of the following dictionary `{"a": {"b": "c"}}` is `("a", "b")`.
In lots of extraction tasks, we want to extract a variable number of JSON objects. The user specifies this with an `"array"` type in the JSON schema. This is often used for large tables — we want to extract many rows from a table that should respect a certain schema.
The issue here is that one small mistake could cost nearly an entire table's worth in score when we score naively. For example, on a table with 100 rows, if the model misses the first row, each predicted row will be positionally off-by-one, leading to a terrible score even though the model effectively got 99/100 correct.
### Our scorer
fig. 4 — The scorer, step by step.
Our scorer is deterministic and has a simple design. It has three components:
- Document normalization;
- Scalar normalization; and
- Pairing leaf addresses:
- If unambiguous, pair exactly; and
- If ambiguous, Hungarian match based on content (recursively in case of nested arrays of objects).
And it works in the following stages:
1. Normalize document;
2. Flatten prediction and gold JSON dictionary to addresses mapped to their scalar values;
3. Normalize scalar values of the flattened addresses; and
4. For each array that appears, Hungarian match (recursively for nested arrays) based on array element content to align ambiguous predicted and gold addresses. There may be unmatched predicted addresses — false positives — and unmatched gold addresses — false negatives.
For each document, this process produces one `Verdict` per unique scalar address, aligned via Hungarian matching when needed. The options are:
- `matched`: was matched and the values match;
- `misread`: was matched and the values don't match;
- `unfound`: ground truth has the address, prediction doesn't;
- `fabricated`: schema offered the address, ground truth is silent but prediction exists;
- `invented_item`: an array element's scalar prediction that paired with nothing; or
- `invented_field`: an address the schema never declared.
These are mutually exclusive in our code and also semantically. The one interesting judgement call we made here is that an address falls under `invented_item` if it falls within an unpaired item (i.e. row), even if the address was an invented field _within_ that array element's schema. We think this is the right call: it signals that this was counted against the model for inventing an item. Addresses outside of arrays that the schema never declared are `invented_field`.
fig. 5 — How each model fell short of 100%.
### How we handle nulls
One inconsistency across scoring harnesses is how they handle `null`. The challenge is how to know when a model is omitting something versus asserting something. This is the difference between saying a cell in a table doesn't have a value versus the cell containing the text `"n/a"`.
We handle nulls in the following way (for both ground truth and prediction):
- We flatten both JSONs to get a mapping from addresses to their values; and
- We inspect the value and ask if it states nothing (omission) or not (assertion):
- `""`, `None`, `"\t\n"` all get turned to `None`; and
- `"NA"`, `"-"`, `"--"` all remain.
For every address that has a `None` value after these steps, _drop the entire address_. This means that omitting an address is the same as having the address point to `None`. We think this is the right approach since, otherwise, the metric can be gamed via schema width.
The gaming would be as follows: add hundreds of optional fields to the ground-truth schema without them actually existing in the benchmark's documents. The models then get awarded for finding nothing by getting (basically) free matches between `None` and `None`, arbitrarily inflating scores.
### Metrics
All metric formulas are based on verdict count, which itself is an address count after alignment. The number of all verdicts is `total`.
```text
shared = |matched|
+ |misread|
|gold| = |matched|
+ |misread|
+ |unfound|
|pred| = |matched|
+ |misread|
+ |fabricated|
+ |invented_item|
+ |invented_field|
total = |matched|
+ |misread|
+ |unfound|
+ |fabricated|
+ |invented_item|
+ |invented_field|
```
The headline metrics are:
- `accuracy` → `|matched| / total`;
- `recall` → `|matched| / |gold|`; and
- `precision` → `|matched| / |pred|`.
### The power of verdicts
What the `Verdict`s allow us to do is provide auditability through _results_, versus just the code that led to those results. Metrics decompose into counts of `Verdict`s, so what you see at the top can be interrogated through a ladder of abstraction, which helps interpretability.
One thing we're building is a small UI that breaks down results by `Verdict` and lets you cross-reference this alongside the PDF and schema. Here's a preview:
fig. 6 — Verdict UI preview, shown mid-development against an earlier corpus snapshot.
## What's next
We look forward to receiving community feedback and continuing to refine the benchmark, add more tooling and build out OmniExtractBench further. In the coming weeks, we'll talk more about some of the issues we see with other benchmarks. We'll also launch tools to help you evaluate extraction results and build confidence.
As always, the benchmark that matters most is the one you run on your own documents — point our [playground](/playground) at your ugliest files and check our numbers against your own. If you have feedback on the benchmark, email us at [hi@datalab.to](mailto:hi@datalab.to).
