# Datalab leads another competitor's extraction benchmark > The benchmark shipped with major scoring bugs. After fixing them, we score 93.6% on the benchmark, leading all other contenders. We encourage you to run your own evals, and not just trust vendor benchmarks. - Canonical: https://www.datalab.to/blog/extractbench-scoring-bug - Published: 2026-08-13 - Authors: Vik Paruchuri LlamaIndex, another document parsing company, recently published **ExtractBench**, a 370-document benchmark for structured extraction, with an accompanying paper and public leaderboard. Its headline metric is a "unified value F1" — the share of leaf values in the target schema you get right, matched against a ground-truth JSON. They scored **Datalab at 64.5%** — rank 13 of 14, near the very bottom. That number was low enough to be suspicious, so we pulled the benchmark, ran it ourselves, and read the grader. Most of that gap turned out to be a **bug in the benchmark's Datalab adapter** — not our extraction. Corrected, Datalab scores **~94%**, behind only LlamaIndex (the creators of the benchmark). Here's what we found.

Unified value-F1 on the 370-document ExtractBench set. Most of the published 64.5% gap is a grader artifact (isolated to 0.73→0.93 in a controlled test below).

## The bug: our confidence metadata was counted as wrong values Datalab's extraction API returns more than the values you asked for. Alongside each field it attaches a **`_meta` sidecar** — a small object with the field's confidence score and a citation back to where in the document the value came from. A field named `total_revenue` ships next to a `total_revenue_meta` sidecar. It's metadata _about_ the extraction, not part of the extracted data. Every extraction harness has to strip that kind of housekeeping key before grading, and ExtractBench's Datalab adapter does — it has a routine that removes `*_citations` and `*_score` keys before the result reaches the scorer. It just **doesn't strip `*_meta`.** So every confidence/citation sidecar we return survives into the graded object, gets lined up against the ground-truth schema (which has no such keys), and is counted as an **extra, wrong predicted value.** The bug's fingerprint is sitting on the leaderboard in plain sight. Datalab has two entries on ExtractBench: our API at **64.5%** (13th), and `lift`, our open-source 9B extraction model, at **77.3%** (10th). `lift` is a bare model — it returns the values you asked for and nothing else, none of the confidence-and-citation `_meta` sidecars our API attaches — so it hands the adapter nothing extra to miscount. The API carries that metadata and gets taxed for every field of it. The result is a tell you can read straight off their own board: it ranks our **free** model roughly **13 points above our paid API**, on the same documents. LlamaIndex not even taking the time to do basic sanity checks on benchmark numbers tells you a lot. The effect is easiest to see on a document we get _perfectly_ right.

What we returned — an earnings deck, every value byte-identical to ground truth

{
  "total_revenue": 1382.0,                              ✓ correct
  "total_revenue_meta": { "confidence": 0.98, "cite": … },   ✗ counted as a wrong value
  "gross_profit": 812.4,                               ✓ correct
  "gross_profit_meta": { "confidence": 0.97, "cite": … },    ✗ counted as a wrong value
  … 14 real KPIs, each with its _meta twin …
}

Every value correct; both segments (revenue and profit) exact. Because the grader kept our 14 _meta sidecars, this scored 0.588 — 20 real values weighed against 20 correct plus 14 metadata artifacts.

A byte-perfect extraction scored **0.588**. And the damage is **systematic** in a way that quietly skews the whole leaderboard. The `_meta` penalty is a roughly fixed tax — one junk key per field — so it hurts most on documents with _few_ real values. Dense tables with thousands of cells barely notice it; sparse documents — single-page forms, a handful of scalar KPIs, tax documents — get cratered. ExtractBench is heavy on exactly those sparse documents, so the aggregate isn't just noisy, it's biased downward: | Document | Real value cells | Correct? | Value-F1 as scored | | ----------------------- | ---------------- | -------- | ------------------ | | Earnings deck (14 KPIs) | 20 | 100% | **0.588** | | Redacted form | 192 | ~correct | 0.54 | | Financial summary | 467 | ~correct | 0.66 | | Dense data table | 8,565 | ~correct | 0.999 | Same extraction quality, wildly different scores — sorted entirely by how much a fixed metadata tax gets to dominate. **How much does this cost us?** The clean way to measure it is a controlled toggle: take the _same_ extraction outputs and score them with `_meta` present versus stripped, changing nothing else. That lifts the score from **0.73 to 0.93** — and on documents we extract perfectly, it's the entire gap (the earnings deck above: **0.588 → 1.000**). Across the full 370-document set, teaching the adapter to strip `_meta` (exactly as it already strips `_citations` and `_score`) brings our corrected aggregate to **91.3%**, versus the **64.5%** LlamaIndex published. ## The second bug: one vendor's format becomes the definition of "correct" Fixing the metadata bug gets us to 91.3%. The rest of the gap is subtler but still structural. Many values have more than one correct representation — a blank numeric field is `0` or `null`, an unchecked box is `false` or `null`, a thousand dollars is `$1,000` or `1000`, fifty percent is `50%` or `0.5`. The benchmark picks **one** of those surface forms for the ground truth (designed around how LlamaIndex returns the values) and scores everything else by exact string match:

Case	Ground truth	Our answer	Graded
Blank numeric field	0	null	wrong
Unchecked checkbox	false	null	wrong
Currency	$1,000	1000	wrong
Percent	50%	0.5	wrong
Whitespace / punctuation	"3, 4"	"3,4"	wrong

Every one of these answers is correct. The answer key just encodes a different surface form — and the benchmark's own scorer notes that typed comparators aren't applied, so it falls back to string comparison and $1,000 ≠ 1000.

It's tempting to file these under "judgment calls." They aren't — because the grading isn't neutral. The party that wrote the answer key also ships the system that tops the leaderboard, and exact-match scoring rewards whichever provider emits the answer key's exact surface form. A system tuned to produce the ground truth's conventions scores full marks; every competitor that normalizes differently is marked wrong for a formatting choice, not an error. That turns one vendor's output format into the definition of "correct" — a structural penalty applied to everyone else, on cells where nobody is actually wrong. The unchecked-checkbox convention alone is 381 fields across 52 tax documents. Score these the way any consumer of the data would — a number is a number regardless of the currency symbol, a blank is a blank, an unchecked box is unchecked — and Datalab goes from 91.3% to **93.6%**. | | Value-F1 (n=370) | | ------------------------------------------ | ---------------- | | Published (LlamaIndex's run, older config) | 64.5% | | `_meta` stripped (current run) | 91.3% | | + format-agnostic matching | **93.6%** | | LlamaExtract Agentic Plus (leader) | 95.6% | By group, the corrected number holds up across document lengths: short **94.1%**, medium **92.2%**, long **94.7%**. ## The usual caveats about vendor benchmarks We'd give this the same treatment we'd give any self-published benchmark — [including our own](/blog/trusting-vendor-benchmarks). ExtractBench was built by LlamaIndex, and LlamaIndex's own product tops it. The party that ranks first also chose the corpus, wrote the schemas, and built the grader (including the adapter that scored everyone else's output). They also used this for significant marketing value _without even doing basic validation on the numbers_. It's just the structure of a benchmark a company publishes about itself, and it's exactly why a metadata-stripping gap in the harness's own Datalab adapter can ship without anyone noticing it was counting our confidence metadata as wrong answers. A few honest notes on our numbers, too: - **Extraction is nondeterministic**, so a single run wobbles by roughly ±0.5–1 point at the aggregate. 93.6% is one run; treat it as ~94%, not a decimal-precise claim. - The `_meta` fix is one line in the harness's Datalab adapter — strip `_meta` alongside `_citations`/`_score` — and it under-counts _any_ provider that returns confidence metadata, not just us. Our current API still emits `_meta`, so the harness under-counts it today. So take our corrected 94% for what it is: strong evidence on a benchmark we didn't design, once a grader bug is removed. For your own decision, the only benchmark that counts is the one you run on your own documents. Point our [playground](/playground) at a few of your ugliest files, define what "correct" means for _your_ workload, and trust what you measure over what any of us publish.