How to Evaluate Document Parsers on Complex PDFs

A parser can recover every word in a table and still put a value under the wrong heading or attach an item to the wrong parent. Evaluating only text similarity misses those structural errors.

This guide builds a small, repeatable check for named fields, table relationships and reading order. It complements the document extraction software evaluation guide, which covers tool-selection criteria. The supplied candidate output is deliberately flawed; it is not a benchmark result for any product.

Start With Reviewed Expectations

Choose representative documents and define the expected output before looking at the parser results. Include the scans, multi-column pages, tables and document types your application actually receives. Keep versions, settings and ground truth alongside each run. Avoid putting confidential documents into shared benchmark fixtures.

The downloadable Python project includes a small illustrative expected record and a deliberately incorrect candidate. To run the checker:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m unittest -v
python evaluate.py expected.json candidate.json

The evaluator reports 2/2 exact field matches, but flags a missing row under A-101 and an unexpected row under B-202: the candidate attached SKU-2 to the wrong order. It also reports 1/2 adjacent reading-order pairs correct. This illustrates why a correct value alone is not enough. The diagram shows additional measurements, such as character error rate and bounding-box overlap, that the small sample checker does not implement.

Normalize Before Comparing

The example expects the same JSON shape from every candidate: fields, rows and reading_order. Write an adapter for each parser output and inspect it before using the scores. An adapter can introduce errors of its own; preserve the original output so you can trace a failure back to the parser or the adapter.

Rows are compared as tuples including the parent ID. The sample also counts duplicates and unexpected rows. Reading order compares each adjacent pair in the expected list; it is a simple diagnostic, not a comprehensive document-layout metric. Add domain-specific checks for merged headers, footnotes, units or chart values when those matter.

Compare Systems Fairly

Use the same source files, expected answers and success criteria for each parser. Record failures, timeouts and configuration rather than dropping difficult pages. Report per-document and per-category results, not just an overall average, and review disagreements against the source.

For extraction workflows, evaluate field values separately from citation locations: a citation can help a reviewer find evidence without proving that a value is correct. Include review effort and operating costs in the decision. Public benchmarks can provide useful context, but a small synthetic example cannot establish how any parser performs on a production workload.

Unsiloed provides Parse and schema-based Extract, so you can evaluate output structure and source citations against your own documents. Create an account to try it.