# We lead an external extraction benchmark (but run your own evals) > A competitor commissioned a long-table extraction benchmark, and scored Datalab at 34% recall. We rebuilt our extraction and now have the top recall (99.1%) and lead on precision too. Still, you should take all vendor benchmarks with a grain of salt and run your own evals. - Canonical: https://www.datalab.to/blog/trusting-vendor-benchmarks - Published: 2026-07-14 - Authors: Vik Paruchuri Recently, a competitor of ours (Reducto) commissioned **LongExtractionBench**, a benchmark for pulling structured data out of long, dense tables. Reducto wrote the grading methodology, and when it launched they were the top performer. They chose recall as their headline metric, and scored Datalab at **33.8%**. Recall here is defined as the share of a document's rows a tool actually captures. This benchmark tested an edge case that we weren't well set up to handle. Like most vendor benchmarks, it was specifically designed for marketing - Reducto was intended to win, and structurally it is very hard for anyone else to match their score. The task, extracting thousands of rows from dense tables, is a valid extraction task, but only matters for a small fraction of customers. Nonetheless, we took this as a challenge. We rebuilt our long-table extraction from the ground up - and today, on the public benchmark set Reducto released (50 documents), **Datalab has the highest recall, ahead of Reducto itself.** For several reasons, we wouldn't read too much into this benchmark, though (more below) - we'd encourage you not to trust vendor benchmarks (including ours!), and to run your own evals. ## We now lead on recall and precision Here's where things stand today on the born-digital set:

Datalab vs Reducto on 50 born-digital docs — our re-run, same corpus and grader (0–100 scale). Datalab leads recall and precision; Reducto edges leaf.

A few notes: - **Datalab and Reducto are a fresh, same-corpus re-run.** We ran both on the same 50 born-digital documents with the official grader, with cache-busting so neither system could serve a stored answer. All three metrics land within a point — these systems are very close. - **Datalab doesn't win every metric.** We lead recall (99.1 vs 98.3) and precision (99.8 vs 99.7); Reducto edges cell-level (leaf) accuracy by ~0.7 (99.8 vs 99.1). We report all three rather than just the one we win. - **On recall, Datalab also leads the rest of the field** — Extend 92.7%, LlamaParse 77.5%, Gemini 3.1 Pro 48.6% (Reducto's published figures on the full 225-document set, which includes documents we can't see; close to what we measure, but not the identical corpus). ## The task it measures LongExtractionBench tests one specific thing: taking a document with a very long, very dense table — up to ~15,000 rows over hundreds of pages — and extracting **every single row** into a JSON schema. Most of the failures in this mode are **recall** failures. It's relatively easy to map table rows to the schema, but doing it perfectly across thousands of rows is challenging. It's even harder if some rows should be included, but not others.

Source — one long, dense table

Timestamp	Channel	Reading	Unit	Flag
09:14:00	CH-07	0.482	V	ok
09:14:01	CH-07	0.483	V	ok
09:14:02	CH-07	0.481	V	ok
⋮ 14,208 rows total

→

Extracted — one object per row

[
  {
    "timestamp": "09:14:00",
    "channel": "CH-07",
    "reading": 0.482,
    "unit": "V",
    "flag": "ok"
  },
  … 14,207 more objects
]

Every row of the table becomes one object in the schema. Recall is the share of all 14,208 rows you actually get back.

## A note on vendor benchmarks LongExtractionBench is **not a general test of document extraction.** It's built around a single edge case — thousands of near-identical rows in dense tables — that most workloads never hit, and that happens to be a Reducto strength. Most documents aren't 15,000-row tables; most extraction jobs are a few dozen fields off an invoice, a form, a contract, or a report, where reading order, layout, handwriting, and language matter far more than raw row throughput. The leader also scores 99+%, which is a red flag that either the benchmark has been fit to the specific vendor (tailored to their strengths), or is relatively easy. Scores on more realistic benchmarks tend to max out at lower scores, with headroom for improvement. Here are some factors to pay attention to in vendor benchmarks:

The corpus who chose the documents — and do they play to your strength?

The metric which number gets to be "the one that matters"?

The field competitors are run once, by the publisher, at settings you can't inspect

Home field the publisher can tune their own system against the eval; nobody else can

Four ways any self-published benchmark tilts toward whoever built it.

Right in the benchmark's limitations section: _"Reducto commissioned this benchmark. Reducto is also one of the systems under test and the top performer on these documents."_ The same party that ranks first also chose the task, commissioned the set, and wrote the grading methodology. This is true for any vendor benchmark, including our own. So take our recall win for exactly what it is: strong evidence on one hard, narrow task, on a benchmark we didn't design. Here's how we recommend you benchmark document extraction systems: 1. **Collect 20–50 documents that look like your real workload** 2. **Define what "correct" means for you.** Every row of a table? A handful of header fields? Reading order? Non-English text? The answer is workload-specific, and it's rarely a single composite score. 3. **Run every candidate the same way**, on the same inputs, and read the actual failures — not just the aggregate. We try to be as objective as possible with our own benchmarking - use external benchmarks where possible, if we have internal benchmarks make them fair and representative of difficult real-world tasks. But you should still run evals on your own documents. We built [Forge Evals](/blog/introducing-forge-evals) for exactly this, and our [playground](/playground) is easy to do some quick testing with. Also, feel free to reach out to us if you have custom eval needs. Use what we offer, use a competitor's tools too, and trust what you measure over what any of us publish.