TL;DR
This paper reframes the task of evidence selection for fact verification from a binary "select/don't select" decision into a user-centric ranking problem. By ordering candidate evidence sentences so that a minimally sufficient set appears as early as possible, the proposed framework minimizes user reading effort while maintaining verification accuracy. The authors demonstrate that an incremental ranking approach powered by Large Language Models (LLMs) or Large Reasoning Models (LRMs) significantly outperforms traditional one-shot ranking and binary selection methods.
What Problem Does This Paper Address?
When automated systems or LLMs verify claims or attribute sources, they typically perform evidence selection—outputting a fixed, unordered subset of sentences. This introduces a fundamental trade-off: optimizing for high precision risks omitting crucial context, while optimizing for high recall overwhelms users with redundant or irrelevant information.
In practice, humans verify claims sequentially, reading until they reach a point of "minimal sufficiency"—the exact moment they have gathered enough information to confidently support or refute a claim. To align with this behavior, the authors introduce the Evidence Ranking task. The goal is to globally rank all candidate sentences so that a sufficient set of evidence surfaces at the very top of the list, allowing users to stop reading as early as possible.
Methods
The authors formalize the task by defining the Minimal Sufficient Rank (MSR)—the smallest prefix in a ranked list that contains a valid, complete set of evidence. To evaluate this, they adapt standard Information Retrieval (IR) metrics:
- Modified Mean Reciprocal Rank (MRR): Measures efficiency by normalizing the observed rank against the Ideal Minimal Sufficient Rank (IMSR).
- Success Rate (SR): A strict metric assessing whether sufficiency is achieved exactly at the theoretically optimal position.
They evaluate two primary algorithmic paradigms across four model families:
- One-Shot Ranking: Ranks all candidate sentences in a single forward pass.
- Incremental Ranking: Iteratively builds the ranking by selecting the next best sentence conditioned on the context of previously selected sentences. This explicitly helps the model capture complementary information and avoid redundancy.
The model families tested include embedding-based similarity (using bge-large-en-v1.5), fine-tuned Natural Language Inference (NLI) models (DeBERTa-v3-large), a task-specific fine-tuned reranker (ReasonRank-7B), and state-of-the-art LLMs/LRMs (GPT-4o, Qwen3, OpenAI o3).
Datasets
Because no dedicated dataset existed for this sequential task, the authors constructed a unified benchmark by aggregating and adapting three prominent fact-verification datasets:
- FEVER: Focuses primarily on single-hop evidence with multiple valid gold evidence sets.
- HoVer: Designed for multi-hop reasoning, requiring longer chains of complementary sentences (2–6 hops) to verify a claim.
- WICE: Features naturally occurring Wikipedia claims with heterogeneous, real-world source text from Common Crawl.
The resulting benchmark consists of 1,000 curated instances (400 FEVER, 400 HoVer, 200 WICE) spanning single-hop, multi-hop, and partially supported claims.
Key Insights and Ideas
- Context is King (Why Incremental Wins): Incremental strategies consistently outperform one-shot approaches. By feeding previously selected sentences back into the prompt, models can recognize what information has already been "covered," effectively filtering out redundant sentences that look superficially relevant but add no new diagnostic value.
- The Failure of NLI on Multi-Sentence Evidence: Fine-tuned NLI models performed poorly on this task (ranking below simple embedding similarity). Because they are trained on single-premise pairs, they struggle to assign high scores to partially supporting sentences, failing to recognize that two individually weak sentences can be collectively sufficient.
- Generalization Trumps Specialization: The task-specific, fine-tuned
ReasonRank-7Breranker was competitive with one-shot LLMs but consistently fell short of general-purpose LLMs operating in an incremental mode. General reasoning capability appears more critical for evaluating evidence complementarity than specialized listwise ranking optimization.
Key Results
- LLM Performance: Incremental LLM ranking using GPT-4o or Qwen3-235B achieved the best baseline results, both reaching an MRR of 0.75 and a Success Rate of ~62%.
- Reasoning Models (LRMs): When evaluating newer reasoning-optimized models on a dataset subset, OpenAI’s o3 achieved a massive performance leap, hitting an MRR of 0.88 and a Success Rate of 79.4%.
- User Study: A controlled study with human verifiers comparing Evidence Ranking (GPT-4o incremental) against standard Evidence Selection demonstrated clear practical benefits:
- Higher Accuracy: Verification success rate jumped from 74% (selection) to 94% (ranking) because selection sets were frequently insufficient.
- Reduced Reading Burden: Users read an average of only 2.48 sentences with ranking, compared to 3.85 sentences with selection.
- Lower Over-reading: Instances where users read more sentences than necessary dropped from 35% to 10%.
Limitations and Future Work
While highly effective, the authors note a few limitations:
- Lack of Coherence Modeling: The underlying datasets do not provide a natural narrative or logical flow for the sentences. Future work should explore how to balance minimal sufficiency with readability and conversational coherence.
- Claim Complexity: The benchmark primarily targets short, focused claims. Extending this to long-form document generation or complex, multi-aspect claims remains an open challenge.
- Conflicting Evidence: The current framework assumes evidence either supports or refutes a claim. Real-world scenarios often contain mixed or contradictory evidence from different sources, which requires more nuanced ranking strategies.
Summary updated