TL;DR
RankShift reveals that document parsing leaderboards are surprisingly fragile: simple changes in how you match text or which ground-truth format you use can flip the rankings of nearly a third of all model pairs. The study advocates for "robustness profiles" rather than single headline scores to truly understand model capabilities.
What Problem Does This Paper Address?
In the world of document parsing—converting images of pages into structured text like Markdown or JSON—we rely heavily on benchmarks like OmniDocBench to tell us which models are "the best." However, these benchmarks aren't just collections of data; they are complex pipelines involving specific design choices: how do you segment the text? How do you match a model's prediction to the ground truth? Which string-similarity metric do you use?
The authors of RankShift argue that these choices are often arbitrary and understudied. If a model's rank changes significantly just because you switched from scoring against a JSON file to scoring against a Markdown file, then the leaderboard isn't just measuring "intelligence"—it's measuring how well a model aligns with a specific, potentially biased, evaluation script.
Methods
The researchers introduced a systematic framework to evaluate ranking instability across three dimensions:
- Dataset Composition: Comparing performance across different page types (e.g., academic papers vs. newspapers) and layouts.
- Alignment Strategy: This is the "how" of matching. They tested 6 configurations, crossing three matching methods (one-to-one matching, adaptive matching, and whole-page concatenation) with two ground-truth formats (JSON vs. Markdown).
- Scoring Metric: Comparing standard NLP metrics like Normalized Edit Distance (NED), BLEU, and METEOR.
To quantify how much the leaderboard "shifted," they used three stability measurements: Kendall’s $\tau$ (global rank correlation), top-$k$ churn (how many models drop out of the top tier), and pairwise reversal rates (how often Model A beats Model B in one setting but loses in another).
Datasets
The study utilized two primary datasets:
- OmniDocBench v1.5: The current "gold standard" for document parsing, containing 1,355 pages across various document types and layouts.
- Real5-OmniDocBench (Scanning): A unique dataset consisting of physical, scanned reconstructions of the exact same pages in OmniDocBench. This allowed the authors to isolate the effect of "capture condition" (digital vs. scan) on model rankings.
Key Insights and Ideas
- The "Matching-Format" Trap: The most shocking finding was that the interaction between the matching algorithm and the ground-truth format is a massive driver of instability. Under segment-level matching, the correlation between rankings on JSON vs. Markdown was a dismal $\tau \approx 0.33$.
- Scans are More Stable than Scripts: Surprisingly, model rankings were more consistent when moving from digital PDFs to physical scans ($\tau = 0.80$) than they were when simply changing the internal matching strategy of the benchmark.
- Metrics are the Most Stable: While researchers often argue over BLEU vs. METEOR, this study found that metric choice is actually the most stable dimension ($\tau \approx 0.78$). If your model is winning on NED, it’s likely winning on BLEU too.
- Hidden Strengths: Aggregate leaderboards hide specialized capabilities. For example, the model
Dolphin 1.5ranks 11th overall but jumps to 4th place when evaluated specifically on complex three-column layouts.
Key Results
The quantitative results highlight a high degree of volatility in current evaluation practices:
- Rank Reversals: Changing the ground-truth format under segment-level matching reversed the relative ordering of 35.8% of model pairs.
- Top-5 Churn: In the most unstable comparison, 3 out of the top 5 models changed when the evaluation strategy was tweaked.
- Alignment Signatures: The authors proposed a diagnostic protocol using "alignment signatures" to identify failure modes. They found that some models are unfairly penalized by specific formats because of "formatting overhead" (like extra Markdown syntax) rather than actual text errors.
Detailed correlation scores across all dimensions are summarized in the table below:
Limitations and Future Work
The authors acknowledge that their diagnostic protocol—which maps specific score patterns to failure modes like "hallucinations"—has not yet been validated against a manually labeled "gold standard" of error types. Furthermore, the study is centered on the OmniDocBench ecosystem. While OmniDocBench is influential, other benchmarks may have different idiosyncratic biases.
The paper concludes with a call to action for the community: leaderboards should stop reporting single, "objective" scores and start providing rank-stability profiles. This would allow practitioners to choose models that are robust across multiple reasonable evaluation designs, rather than those that have simply overfitted to a specific benchmark's quirks.
Summary updated