Unsiloed vs Unstract on olmOCR-Bench: 88.5% vs 13.7%

Published by Unsiloed AI | July 2026

We already published a hands-on comparison of Unsiloed and Unstract's LLMWhisperer on four documents we chose ourselves. The obvious objection to that kind of test is that we picked the documents. So we ran the same two systems against a public benchmark that neither of us designed, and scored both with a scorer we didn't write.

The sample: 75 documents drawn from Allen AI's olmOCR-Bench, spread across all 7 of its categories, carrying 452 individual pass/fail tests. The scorer: the unmodified upstream olmocr==0.4.27 test suite. No custom judging, no LLM rescue pass, no rule adjustments.

The result: Unsiloed Parser passed 400 of 452 tests (88.5%). Unstract LLMWhisperer passed 62 (13.7%).

For context we scored four widely-used vision-LLM parsers on the identical subset. Unsiloed led all of them too, by 3.5 to 9.3 percentage points.

Rank	System	Tests passed	Pass rate
1	Unsiloed Parser	400 / 452	88.5%
2	GPT-5.5	384 / 452	85.0%
3	GPT-5.6 Sol	382 / 452	84.5%
4	Kimi K3	379 / 452	83.8%
5	Gemini 3.1 Pro	358 / 452	79.2%
6	Unstract LLMWhisperer	62 / 452	13.7%

A 75-point gap is not a normal benchmark result, and it isn't the story of one system reading characters six times better than another. It's the story of two products that were built to return different things. The rest of this post is about why the gap is that shape.

How We Ran It

• Source dataset: olmOCR-Bench, 1,403 single-page PDFs across 7 real-world categories — math-heavy academic papers, headers and footers, small print, multi-column layouts, degraded historical scans, math inside historical scans, and complex tables.
• Selection: 7–14 randomly shuffled documents per category, seeded for reproducibility, so no category dominates. We started from 84 documents and dropped the handful where either pipeline hit a transient error, leaving 75 documents with clean results from both systems.
• Category counts: arXiv math (14), headers/footers (13), long tiny text (7), multi-column (12), old scans (11), old scans with math (11), complex tables (7).
• Tests: each document carries several tests — a specific equation transcribed, a specific string appearing before another, a specific table cell adjacent to another. 452 in total across the 75 documents.
• Modes: for LLMWhisperer we used the mode that gives it the best chance on each category rather than the default everywhere. On multi-column pages that meant flat text, since layout_preserving interleaves the columns.
• Freshness: both sets of outputs were generated on 30–31 July 2026 against the current production version of each service.

Per-Category Results

Category	Tests	Unsiloed	Unstract	Vision-LLM avg	vs Unstract	vs best LLM
Complex tables	33	90.9%	0%	73.5%	+90.9 pp	+15.1 pp (Kimi 75.8%)
Headers & footers	42	100%	38.1%	91.7%	+61.9 pp	+2.4 pp (GPT-5.5/5.6 97.6%)
Old scans with math	140	91.4%	0%	83.6%	+91.4 pp	+3.5 pp (GPT-5.5 87.9%)
ArXiv math papers	74	93.2%	0%	92.6%	+93.2 pp	−4.1 pp (GPT-5.5 97.3%)
Long tiny text	51	96.1%	15.7%	94.6%	+80.4 pp	tied (Kimi 96.1%)
Multi-column layouts	50	82%	20%	77%	+62 pp	+2 pp (Kimi/Gemini 80%)
Old handwritten scans	62	66.1%	45.2%	65.7%	+20.9 pp	−1.6 pp (Kimi 67.7%)
Overall	452	88.5%	13.7%	83.1%	+74.8 pp	+3.5 pp (GPT-5.5)

Unsiloed leads 5 of 7 categories outright, ties Kimi K3 on long tiny text, and finishes within 4 points of first on the other two. Against Unstract it wins every category, by margins from 20 pp on handwritten scans to 93 pp on arXiv math.

Three of those zeros are worth pausing on. They are not the scorer being unfair.

Where the 0% Categories Come From

247 of the 452 tests require structured output — LaTeX for an equation, or a table with addressable cells. LLMWhisperer scored zero on all of them, and it does so by design: it returns plain text and expects a language model you supply to do the structuring.

Tables (0 / 33)

Table tests check relationships: "cell X sits directly above cell Y", "cell Z is the leftmost value in its row". Answering them requires a two-dimensional structure.

Unstract returns an ASCII grid:

+--------------------------------------------------------+--------------------+---------------------------------------------------+
| Milestone                                              | Milestone Date     | Deliverable                                       |
+--------------------------------------------------------+--------------------+---------------------------------------------------+

Unsiloed returns a markdown pipe table:

| Milestone                   | Milestone Date     | Deliverable                             |
|-----------------------------|--------------------|-----------------------------------------|
| Initiate Pre-design         | September 10, 2001 | Notice to Proceed                       |
| Complete Pre-design         | March 10, 2002     | Pre-design report, submitted to EPA/EPD |

Both are readable by a human. Only one is queryable by a program. To the scorer, the ASCII grid is prose containing pipe characters, so no cell relationship can be verified and all 33 tests fail. Unsiloed's 30/33 in this category was also the highest of the seven systems, 15 points ahead of Kimi K3 (75.8%).

Math, clean and degraded (0 / 214)

Math tests compare a rendered LaTeX expression against a gold expression. Unsiloed emits \frac{a}{b} and \int; LLMWhisperer emits a best-effort plain-text rendering — a/b, "integral of". There is nothing for the scorer to render, so every one of the 140 old-scans-math and 74 arXiv-math tests fails.

This is the one place where the benchmark flatters a frontier vision-LLM over us: GPT-5.5 takes clean arXiv math 97.3% to our 93.2%. On the harder variant — the same notation on degraded historical scans, where the OCR has to be right before the LaTeX can be — Unsiloed leads the field at 91.4%.

Headers and footers (38.1%), and the value of a 100%

Unsiloed was the only system in the comparison to score a perfect 100% here, ahead of GPT-5.5 and GPT-5.6 Sol at 97.6%.

Half of these tests are absence tests: a running head or page number should not appear in the body text. Unsiloed classifies page chrome as a distinct region during layout analysis, so both the presence and absence tests resolve. LLMWhisperer returns the page as one text stream, so it passes the presence tests and fails the absence ones almost mechanically — which is roughly what 38.1% looks like.

If you're building a RAG index, this category is the one that predicts how much junk ends up in your chunks.

Where Unstract Does Well

Old handwritten scans, 45.2% — its strongest category, and not by accident. These tests mostly ask whether a string of text is present, and structural markup barely matters. On pure character recognition over damaged pages, LLMWhisperer is competitive with far more elaborate pipelines. It is also the category where every system struggles: Unsiloed managed 66.1%, Kimi K3 67.7%, and the vision-LLM average was 65.7%.

That's the honest read of the 75-point overall gap. It is not evidence that LLMWhisperer's OCR is six times worse. On a benchmark that measured word error rate alone, the two systems would look far closer. olmOCR-Bench doesn't measure that; it measures whether the output is a structured representation of the page.

Why the Architectures Produce These Numbers

Both systems accept a PDF page and return text. What happens in between explains everything above.

Unsiloed Parser treats a page as a set of regions understood in relation to each other:

1. Layout analysis — a vision model finds the regions before any character recognition begins: text blocks, tables, figures, section headers, formulas, page chrome.
2. Region-specialised transcription — each region goes to the engine suited to its content type; a table specialist reads tables, a formula-aware model reads equations.
3. Region-specialised structuring — prose becomes markdown, tables become markdown pipe tables, equations become LaTeX, HTML where markup fidelity matters.
4. Reading-order reconstruction — a final pass orders the structured regions the way a human reads them, handling multi-column layouts, wrapped columns and interleaved figures.

Unstract LLMWhisperer treats the page as a single unit to transcribe: pick a mode (native text for digital PDFs, OCR for scans), recognise characters, optionally insert whitespace to approximate the geometry, return the text.

Unstract is explicit about this. Its documentation positions layout-preserving mode as a way to help a downstream LLM read the document, not as a document-understanding product in itself. If you want markdown headings, LaTeX or structured tables, you're expected to call a model after LLMWhisperer to produce them. Unsiloed builds that step into the pipeline.

Which is also why the multi-column result lands where it does. LLMWhisperer's layout-preserving mode is a rendering: column one and column two share the same physical lines, so any consumer that normalises whitespace — most chunkers, most embedding pipelines — collapses the page back into interleaved fragments. Even in flat-text mode, which we used here to give it the best shot, there's no explicit reading-order logic. It scored 20%; Unsiloed's dedicated reading-order pass scored 82%, the highest in the field.

One Page, Side by Side

The arXiv title page (arxiv_math/2503.03772_pg1) shows the difference in a form no score conveys.

Unstract:

Cardinalities in finite monoids of G-equivariant functions
Ramo´nH.Ruiz-Medina*
CentroUniversitariodeCienciasExactaseIngenier´ıas,
UniversidaddeGuadalajara,Guadalajara,Me´xico.

Unsiloed:

Cardinalities in finite monoids of G-equivariant functions

Ramón H. Ruiz-Medina*
Centro Universitario de Ciencias Exactas e Ingenierías,
Universidad de Guadalajara, Guadalajara, México.

Two failure modes compound here. Word boundaries are lost where the source glyphs sit close together, so Centro Universitario de Ciencias Exactas e Ingenierías arrives as a single 49-character token. And accents never compose: ´n stays two characters, so searching that text for "Ramón" or "México" returns nothing.

Neither of those is an OCR miss in the usual sense — every character is there. They're what happens when nothing downstream of recognition puts the text back together.

What This Does and Doesn't Tell You

It doesn't say LLMWhisperer is a bad text extractor. It's fast, typically 1–5 seconds a page, deterministic, cheap, and credible on degraded scans when the test is text presence. If you already have a language model in the loop and only need characters off the page, it does that job.

It does say that if the structured document is the deliverable — invoices to JSON, papers chunked for a RAG index, forms into a database — the gap between the two products is not marginal. On this benchmark it's 400 tests to 62, and 247 of those tests are ones a text-extraction preprocessor cannot answer at all.

The broader point is the one we made when Unsiloed took the top spot on the full olmOCR-Bench: a full-document pipeline beats both a preprocessor and a single-pass frontier vision-LLM on this class of work, and it beats them for a structural reason, not a marginal one.

Reproducing This

Scoring was done with Allen AI's public olmOCR-Bench scorer, unmodified, pinned at olmocr==0.4.27, against the allenai/olmOCR-bench dataset. Per-vendor runner code for our full-benchmark runs is open source at github.com/Unsiloed-AI/unsiloed-olmocr-benchmark.

To see Unsiloed on your own documents instead of ours, upload one at unsiloed.ai/playground.

---

Logos and trademarks are the property of their respective owners. Use does not imply endorsement.

Scores measured on a 75-document / 452-test sample of olmOCR-Bench using the unmodified olmocr==0.4.27 scorer. Outputs generated 30–31 July 2026.