User-Centric Evidence Ranking for Attribution and Fact Verification

Guy Alt, Eran Hirsch, Serwar Basch, Ido Dagan, Oren Glickman

TL;DR

This paper reframes the task of evidence selection for fact verification from a binary "select/don't select" decision into a user-centric ranking problem. By ordering candidate evidence sentences so that a minimally sufficient set appears as early as possible, the proposed framework minimizes user reading effort while maintaining verification accuracy. The authors demonstrate that an incremental ranking approach powered by Large Language Models (LLMs) or Large Reasoning Models (LRMs) significantly outperforms traditional one-shot ranking and binary selection methods.

What Problem Does This Paper Address?

When automated systems or LLMs verify claims or attribute sources, they typically perform evidence selection—outputting a fixed, unordered subset of sentences. This introduces a fundamental trade-off: optimizing for high precision risks omitting crucial context, while optimizing for high recall overwhelms users with redundant or irrelevant information.

In practice, humans verify claims sequentially, reading until they reach a point of "minimal sufficiency"—the exact moment they have gathered enough information to confidently support or refute a claim. To align with this behavior, the authors introduce the Evidence Ranking task. The goal is to globally rank all candidate sentences so that a sufficient set of evidence surfaces at the very top of the list, allowing users to stop reading as early as possible.

Figure 1: Illustration of the Evidence Ranking task. Given a claim and candidate evidence sentences, the system ranks the sentences so that users can stop read- ing once sufficient evidence is observed. Our evaluation simulates this process by measuring the number of sen- tences a user would read. Green sentences denote rele- vant evidence, and yellow denotes redundant evidence that can be skipped.

Methods

The authors formalize the task by defining the Minimal Sufficient Rank (MSR)—the smallest prefix in a ranked list that contains a valid, complete set of evidence. To evaluate this, they adapt standard Information Retrieval (IR) metrics:

  • Modified Mean Reciprocal Rank (MRR): Measures efficiency by normalizing the observed rank against the Ideal Minimal Sufficient Rank (IMSR).
  • Success Rate (SR): A strict metric assessing whether sufficiency is achieved exactly at the theoretically optimal position.

They evaluate two primary algorithmic paradigms across four model families:

  1. One-Shot Ranking: Ranks all candidate sentences in a single forward pass.
  2. Incremental Ranking: Iteratively builds the ranking by selecting the next best sentence conditioned on the context of previously selected sentences. This explicitly helps the model capture complementary information and avoid redundancy.
One-shot vs. Incremental ranking paradigms.

The model families tested include embedding-based similarity (using bge-large-en-v1.5), fine-tuned Natural Language Inference (NLI) models (DeBERTa-v3-large), a task-specific fine-tuned reranker (ReasonRank-7B), and state-of-the-art LLMs/LRMs (GPT-4o, Qwen3, OpenAI o3).

Datasets

Because no dedicated dataset existed for this sequential task, the authors constructed a unified benchmark by aggregating and adapting three prominent fact-verification datasets:

  • FEVER: Focuses primarily on single-hop evidence with multiple valid gold evidence sets.
  • HoVer: Designed for multi-hop reasoning, requiring longer chains of complementary sentences (2–6 hops) to verify a claim.
  • WICE: Features naturally occurring Wikipedia claims with heterogeneous, real-world source text from Common Crawl.

The resulting benchmark consists of 1,000 curated instances (400 FEVER, 400 HoVer, 200 WICE) spanning single-hop, multi-hop, and partially supported claims.

Overview and key statistics of the curated datasets.

Key Insights and Ideas

  • Context is King (Why Incremental Wins): Incremental strategies consistently outperform one-shot approaches. By feeding previously selected sentences back into the prompt, models can recognize what information has already been "covered," effectively filtering out redundant sentences that look superficially relevant but add no new diagnostic value.
  • The Failure of NLI on Multi-Sentence Evidence: Fine-tuned NLI models performed poorly on this task (ranking below simple embedding similarity). Because they are trained on single-premise pairs, they struggle to assign high scores to partially supporting sentences, failing to recognize that two individually weak sentences can be collectively sufficient.
  • Generalization Trumps Specialization: The task-specific, fine-tuned ReasonRank-7B reranker was competitive with one-shot LLMs but consistently fell short of general-purpose LLMs operating in an incremental mode. General reasoning capability appears more critical for evaluating evidence complementarity than specialized listwise ranking optimization.

Key Results

  • LLM Performance: Incremental LLM ranking using GPT-4o or Qwen3-235B achieved the best baseline results, both reaching an MRR of 0.75 and a Success Rate of ~62%.
  • Reasoning Models (LRMs): When evaluating newer reasoning-optimized models on a dataset subset, OpenAI’s o3 achieved a massive performance leap, hitting an MRR of 0.88 and a Success Rate of 79.4%.
Evaluation results across different ranking methods.
  • User Study: A controlled study with human verifiers comparing Evidence Ranking (GPT-4o incremental) against standard Evidence Selection demonstrated clear practical benefits:
  • Higher Accuracy: Verification success rate jumped from 74% (selection) to 94% (ranking) because selection sets were frequently insufficient.
  • Reduced Reading Burden: Users read an average of only 2.48 sentences with ranking, compared to 3.85 sentences with selection.
  • Lower Over-reading: Instances where users read more sentences than necessary dropped from 35% to 10%.

Limitations and Future Work

While highly effective, the authors note a few limitations:

  1. Lack of Coherence Modeling: The underlying datasets do not provide a natural narrative or logical flow for the sentences. Future work should explore how to balance minimal sufficiency with readability and conversational coherence.
  2. Claim Complexity: The benchmark primarily targets short, focused claims. Extending this to long-form document generation or complex, multi-aspect claims remains an open challenge.
  3. Conflicting Evidence: The current framework assumes evidence either supports or refutes a claim. Real-world scenarios often contain mixed or contradictory evidence from different sources, which requires more nuanced ranking strategies.

Summary updated

Abstract

Attribution and fact verification are critical challenges in natural language processing for assessing information reliability. While automated systems and Large Language Models (LLMs) aim to retrieve and select concise evidence to support or refute claims, they often present users with either insufficient or overly redundant information, leading to inefficient and error-prone verification. To address this, we propose Evidence Ranking, a novel task that prioritizes presenting sufficient information as early as possible in a ranked list. This minimizes user reading effort while still making all available evidence accessible for sequential verification. We compare two approaches for the new ranking task: one-shot ranking and incremental ranking. We introduce a new evaluation framework, inspired by information retrieval metrics, and construct a unified benchmark by aggregating existing fact verification datasets. Extensive experiments with diverse models show that incremental ranking strategies better capture complementary evidence and that LLM-based methods outperform shallower baselines, while still facing challenges in balancing sufficiency and redundancy. Compared to evidence selection, we conduct a controlled user study and demonstrate that evidence ranking both reduces reading effort and improves verification. This work provides a foundational step toward more interpretable, efficient, and user-aligned information verification systems.
BibTeX
@inproceedings{alt2026user,
  author = {Guy Alt and Eran Hirsch and Serwar Basch and Ido Dagan and Oren Glickman},
  title = {{User-Centric Evidence Ranking for Attribution and Fact Verification}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.340},
}
Figure