TL;DR
Model rankings in NLP are notoriously fragile, often flipping when minor evaluation details—like the gold standard version or the scoring threshold—are changed. This paper introduces contract-explicit evaluation, a five-step protocol that uses statistical "scoped certificates" to identify which model comparisons are truly robust and which are merely artifacts of a specific evaluation setup.
What Problem Does This Paper Address?
In fields like Document-level Relation Extraction (DocRE) and LLM-as-judge evaluation, we often treat a model's score as an inherent property of the model itself. However, the authors argue that performance readings are actually joint properties of the model and the evaluation contract (the specific combination of gold labels, scoring functions, and thresholds used).
The "gap" is that rankings frequently reverse when you swap one valid contract for another (e.g., moving from the original DocRED dataset to the corrected Re-DocRED). Currently, there is no formal method to determine which leaderboard rankings are "real" and which would vanish if the evaluation settings were slightly tweaked. This leads to "trust inflation," where researchers report high confidence in results that are actually sensitive to arbitrary choices.
Methods
The authors propose a formal, preregistered five-step protocol to move from "auditing" evaluations to issuing "certificates" of robustness:
- Declare: Explicitly define and freeze all evaluation axes (gold versions, scorers, thresholds) before running comparisons.
- Measure: Calculate "flip density"—how often model rankings reverse across different contracts. This is grounded in the classical concordance decomposition of Kendall’s rank correlation ($\tau$).
- Scope: Use a "scoping map" to determine which regions of the evaluation space are stable enough to support statistical guarantees.
- Certify: Issue distribution-free "census certificates" (using Clopper–Pearson confidence intervals) for stable regions.
- Hedge: For comparisons that fall outside the stable scope, the system "hedges" by declaring a tie rather than a potentially false ranking.
Technically, the paper explores a Learned Evaluation Reader—a neural network that weights different evaluation contracts to find a consensus—and wraps it in a conformal decision layer to provide risk-controlled guarantees on the "wrong-commit" rate.
Datasets
The study evaluates two major "ecosystems":
- DocRE (Document-level Relation Extraction): Uses 44 training runs across 7 architecture families. It utilizes DocRED, Re-DocRED, and TACRED (including TACREV and Re-TACRED revisions) to test how rankings change across different versions of "truth."
- LLM-as-Judge: Analyzes a massive census of 49,938 battles involving 17 different LLM judges (derived from Chatbot Arena data).
- WMT19: Used as a control case for exchangeable metric axes.
Key Insights and Ideas
- The "Hedge or Commit" Philosophy: Instead of forcing a ranking, the protocol argues that if valid evaluation contracts disagree, the only honest answer is a tie.
- Scoping Identity: The authors show that certifiability is fundamentally anti-correlated with sensitivity. You cannot "argue" a sensitive result into being robust; you can only widen the certifiable region by improving the consistency of the underlying contracts or increasing the population size.
- Judge Error vs. Ambiguity: In the LLM-judge ecosystem, the authors found that 65% of contested battles are due to judge error, not inherent ambiguity in the prompts. This suggests that "LLM-as-judge" is often more a measure of judge consistency than model quality.
- The Simplification Finding: Interestingly, the authors found that their complex "learned" reader could often be replaced by a simple "unanimity gate" (only ranking models if all contracts agree). The value of the method is the discipline of the protocol, not necessarily the complexity of the weights.
Key Results
- DocRE Audit: In a pool of 44 model runs, the scoring dimension was certified with a Clopper–Pearson upper bound of 0.146. The audit revealed that the entire margin of the calibration dial was essentially one "withdrawn commit," highlighting how thin the line is between a robust result and noise.
- LLM-Judge Census: 12.25% of leaderboard comparisons required hedging (declaring a tie) at the pre-declared split criterion.
- Ranking Stability: By applying a two-tier reading, they could certify 69% of battles at a very tight bound ($\le 0.072$).
- Leaderboard Impact: Hedging the "unstable" third of the LLM leaderboard was found to be "costless" for top-tier decisions—the top models remained the top models, but the noise in the middle of the pack was successfully flagged.
Limitations and Future Work
- Registry Power: The authors admit that small registries (e.g., only 13 flip pairs) have limited statistical power. They provide a power simulation script to help future researchers size their audits correctly.
- Exchangeability Requirement: The protocol relies on the assumption that different contracts are "exchangeable" (reading the same underlying quantity). When this fails (as seen in the TACRED analysis where revisions were actually different tasks), the certificates are withdrawn.
- Future Directions: The authors suggest moving toward learning contract preferences that can transfer across different task families and connecting these stable readings to better downstream model-selection decisions.
For those interested in reproducing or applying this protocol, the authors have released the protocol, certifiability maps, ledgers, and per-instance artifacts on GitHub.
Summary updated