RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing

Halimat Afolabi, Daeyoung LeearXiv 2026 · arXiv

TL;DR

RankShift reveals that document parsing leaderboards are surprisingly fragile: simple changes in how you match text or which ground-truth format you use can flip the rankings of nearly a third of all model pairs. The study advocates for "robustness profiles" rather than single headline scores to truly understand model capabilities.

What Problem Does This Paper Address?

In the world of document parsing—converting images of pages into structured text like Markdown or JSON—we rely heavily on benchmarks like OmniDocBench to tell us which models are "the best." However, these benchmarks aren't just collections of data; they are complex pipelines involving specific design choices: how do you segment the text? How do you match a model's prediction to the ground truth? Which string-similarity metric do you use?

The authors of RankShift argue that these choices are often arbitrary and understudied. If a model's rank changes significantly just because you switched from scoring against a JSON file to scoring against a Markdown file, then the leaderboard isn't just measuring "intelligence"—it's measuring how well a model aligns with a specific, potentially biased, evaluation script.

Methods

The researchers introduced a systematic framework to evaluate ranking instability across three dimensions:

  1. Dataset Composition: Comparing performance across different page types (e.g., academic papers vs. newspapers) and layouts.
  2. Alignment Strategy: This is the "how" of matching. They tested 6 configurations, crossing three matching methods (one-to-one matching, adaptive matching, and whole-page concatenation) with two ground-truth formats (JSON vs. Markdown).
  3. Scoring Metric: Comparing standard NLP metrics like Normalized Edit Distance (NED), BLEU, and METEOR.

To quantify how much the leaderboard "shifted," they used three stability measurements: Kendall’s $\tau$ (global rank correlation), top-$k$ churn (how many models drop out of the top tier), and pairwise reversal rates (how often Model A beats Model B in one setting but loses in another).

Figure 1: Evaluation dataset, alignment strategy and scoring metric are varied independently, and the resulting leaderboards compared using Kendall’s τ, top-k churn, and pairwise reversals.

Datasets

The study utilized two primary datasets:

  • OmniDocBench v1.5: The current "gold standard" for document parsing, containing 1,355 pages across various document types and layouts.
  • Real5-OmniDocBench (Scanning): A unique dataset consisting of physical, scanned reconstructions of the exact same pages in OmniDocBench. This allowed the authors to isolate the effect of "capture condition" (digital vs. scan) on model rankings.

Key Insights and Ideas

  • The "Matching-Format" Trap: The most shocking finding was that the interaction between the matching algorithm and the ground-truth format is a massive driver of instability. Under segment-level matching, the correlation between rankings on JSON vs. Markdown was a dismal $\tau \approx 0.33$.
  • Scans are More Stable than Scripts: Surprisingly, model rankings were more consistent when moving from digital PDFs to physical scans ($\tau = 0.80$) than they were when simply changing the internal matching strategy of the benchmark.
  • Metrics are the Most Stable: While researchers often argue over BLEU vs. METEOR, this study found that metric choice is actually the most stable dimension ($\tau \approx 0.78$). If your model is winning on NED, it’s likely winning on BLEU too.
  • Hidden Strengths: Aggregate leaderboards hide specialized capabilities. For example, the model Dolphin 1.5 ranks 11th overall but jumps to 4th place when evaluated specifically on complex three-column layouts.

Key Results

The quantitative results highlight a high degree of volatility in current evaluation practices:

  • Rank Reversals: Changing the ground-truth format under segment-level matching reversed the relative ordering of 35.8% of model pairs.
  • Top-5 Churn: In the most unstable comparison, 3 out of the top 5 models changed when the evaluation strategy was tweaked.
  • Alignment Signatures: The authors proposed a diagnostic protocol using "alignment signatures" to identify failure modes. They found that some models are unfairly penalized by specific formats because of "formatting overhead" (like extra Markdown syntax) rather than actual text errors.
Figure 2: Experiment 2 bump chart showing rank trajectories under alignment variation.

Detailed correlation scores across all dimensions are summarized in the table below:

Table 2: Pairwise Kendall’s τ between model rankings (mean page-level scores). 95% CIs from bootstrap resampling of models (n = 16), B = 2000 replicates.

Limitations and Future Work

The authors acknowledge that their diagnostic protocol—which maps specific score patterns to failure modes like "hallucinations"—has not yet been validated against a manually labeled "gold standard" of error types. Furthermore, the study is centered on the OmniDocBench ecosystem. While OmniDocBench is influential, other benchmarks may have different idiosyncratic biases.

The paper concludes with a call to action for the community: leaderboards should stop reporting single, "objective" scores and start providing rank-stability profiles. This would allow practitioners to choose models that are robust across multiple reasonable evaluation designs, rather than those that have simply overfitted to a specific benchmark's quirks.

Summary updated

Abstract

Document parsing benchmarks are often treated as stable measures of model quality, yet rankings may depend substantially on benchmark design choices. In this work, we present RankShift, a systematic study comparing OCR and vision-language models (VLMs) under controlled variation in dataset composition, matching strategy, and scoring methodology. Using OmniDocBench as a case study, we evaluate the ranking stability of 16 models across 3 dataset configurations, 6 matching strategies, and 3 scoring metrics. We quantify rank stability using pairwise rank correlation, top-k churn, and pairwise reversal rate. Our results show that the interaction between matching strategy and ground truth format is a substantial but understudied driver of instability: scoring against OmniDocBench's JSON annotations versus its Markdown rendering yields a mean Kendall's tau of approximately 0.33 under region-level matching strategies, with individual pairs as low as tau = 0.23, compared with tau = 0.77 when each page is scored as a single concatenated string. Dataset composition produces comparable instability across page-type strata (mean tau = 0.48) and layout-type strata (mean tau = 0.49), while the cross-dataset shift from OmniDocBench to Real5's Scanning reconstruction of the same pages is more stable (tau = 0.80). Metric choice is the most stable dimension, with mean tau of approximately 0.78 across NED, BLEU, and METEOR. These findings indicate that leaderboard position is not solely a property of model capability, motivating leaderboards that report not only headline scores but also the robustness of rankings across reasonable design choices.
BibTeX
@misc{afolabi2026rankshift,
  author = {Halimat Afolabi and Daeyoung Lee},
  title = {{RankShift: Evaluating Ranking Instability Across Benchmark Dimensions in Document Parsing}},
  year = {2026},
  eprint = {2611.10679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10679},
}
Figure