TL;DR
ParseBench-2 introduces a "severity-aware" framework for evaluating document parsing, ensuring that critical factual errors—like a wrong number in a financial table—are penalized far more heavily than benign formatting changes. By optimizing metrics against expert-defined "severity policies" using coding agents, the authors provide a comprehensive benchmark of 4,317 pages that finally aligns automated scores with human judgment.
What Problem Does This Paper Address?
Document parsing is the unsung hero of modern AI pipelines, turning messy PDFs and images into structured data for RAG systems and agents. However, evaluating these parsers is notoriously difficult. Traditional metrics like edit distance or BLEU are "severity-blind": they treat every character change with equal weight.
On a long page, a single digit change in a million-dollar revenue figure might barely nudge the score, yet it represents a catastrophic failure for a financial analyst. Conversely, a parser might be penalized for using different HTML tags to represent the same table, even if the data is perfectly preserved. We need metrics that care about consequences, not just character overlaps.
Methods
The core innovation is a method for developing metrics that follow a severity policy. Instead of just designing a metric and hoping it works, the authors treat metric design as an optimization problem:
- Define a Severity Policy: Experts categorize errors (Critical, Major, Minor, Benign) and set target score drops for each (e.g., a critical error must drop the score by at least 10 points).
- Create Test Scenarios: They generate thousands of "perturbed" outputs where a single specific error is introduced into a correct parse.
- Optimization Loop: A coding agent (under human review) iteratively refines the metric's implementation—its comparison rules and error detectors—until the metric's score drops match the expert's targets.
The final ParseBench-2 score is the lower of two values: a content recovery score (how much was found) and a severity cap (a penalty based on detected errors). This ensures that "recovering most of a page cannot hide a serious error."
Datasets
The authors introduce ParseBench-2, a massive expansion of their previous work. It contains 4,317 pages across six dimensions of parsing quality:
- Text & Formatting: Including difficult cases like redlined text and strikethroughs.
- Tables: Featuring complex headers and dense financial data.
- Charts: Moving beyond simple point-checking to full data table extraction for 1,025 charts.
- Visual Grounding: Ensuring extracted text is correctly localized on the page.
- Forms: A new addition covering 600 government-form pages from 30 countries.
Key Insights and Ideas
- Meaning over Markup: A good metric should be indifferent to "benign variations." Whether a table is represented as HTML or Markdown shouldn't matter if the facts are the same. ParseBench-2 metrics penalize at most 3.4% of such variations, a huge improvement over existing baselines.
- The "Aha" of the Severity Cap: Traditional averaging washes out critical failures. By taking the
min()of recovery and the severity cap, the benchmark forces parsers to be both comprehensive and factually reliable. - Factual Integrity in Tables: The authors represent table cells as triples:
(row_path, col_path, value). This ensures that a value is only considered "correct" if it's associated with the right headers.
Key Results
The evaluation spanned 44 configurations, including frontier Vision-Language Models (VLMs) and specialized document parsers.
- Specialized Parsers Lead: LlamaParse (Agentic Plus) achieved the highest overall score of 90.1, outperforming the best direct vision model, Claude Opus 5.5 (85.7).
- Alignment with Experts: ParseBench-2 metrics achieved significantly higher agreement with human severity judgments compared to existing benchmarks like OmniDocBench or PubTables-1M.
- Cost vs. Accuracy: The study revealed a 30-fold variance in cost among parsers. Interestingly, higher-priced systems didn't always perform better, though specialized "agentic" modes provided the most significant accuracy gains for a moderate cost increase.
Limitations and Future Work
While comprehensive, the authors note a few areas for growth:
- Diagrams: The current benchmark does not yet score complex diagrams or flowcharts.
- Human Reading Error: For unlabeled charts, the "allowance" for reading errors is estimated based on literature rather than direct measurement on this specific dataset.
- Downstream Impact: A major next step is validating how well these severity-aware scores correlate with the performance of downstream RAG applications and AI agents.
In conclusion, ParseBench-2 moves document parsing evaluation away from simple pattern matching and toward a more nuanced, "human-in-the-loop" understanding of what makes an error truly critical.
Summary updated