From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification

Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

Introduction

Automated Fact Verification (FV) is a critical task in the fight against misinformation, typically structured as a three-stage pipeline: document retrieval, evidence extraction, and claim verification. In this workflow, a system must first identify relevant documents, extract specific sentences as evidence, and finally use a "verifier" model to determine if the claim is Supported, Refuted, or if there is Not Enough Info.

Historically, evidence retrievers have been designed based on the Probability Ranking Principle (PRP), which ranks sentences according to their topical relevance to a claim. However, this approach assumes that human-judged relevance is the optimal signal for machine-driven verification. In practice, a sentence may be topically relevant but contain redundant or conflicting information that confuses the downstream verifier. To bridge this gap, this research introduces the Feedback-based Evidence Retriever (FER), shifting the objective from retrieving "relevant" evidence to retrieving "useful" evidence.

The FER Framework

The authors propose a coarse-to-fine retrieval strategy that incorporates feedback from the verifier to optimize the selection of evidence. The process is divided into two primary stages:

1. Coarse Retrieval

A standard BERT-based model is employed to filter the massive Wikipedia corpus (containing over 5.4 million documents) down to a manageable candidate set of the top 25 sentences.

2. Fine-grained Retrieval with Feedback

This stage represents the core innovation of the paper. The retriever ($R_\theta$) selects a subset of sentences from the candidates using a dual-objective loss function:

  • Evidence Classification Loss ($L_{cla}$): A supervised signal that encourages the model to select known ground-truth evidence.
  • Utility Divergence Loss ($L_{uti}$): This provides the feedback loop. The model measures the divergence between the verifier’s prediction when provided with "perfect" ground-truth evidence versus the prediction it makes using the evidence selected by the retriever.

To ensure the selection process is differentiable—allowing gradients to flow from the verifier back to the retriever—the authors utilize the Gumbel Softmax trick. This allows the retriever to learn which specific combinations of sentences maximize the verifier's performance, effectively treating the verifier as a teacher that evaluates the retriever's output.

Experimental Setup and Results

The model was evaluated using the FEVER (Fact Extraction and VERification) dataset, the industry standard for fact-checking tasks. Performance was measured using Information Retrieval (IR) metrics (Precision, Recall, and F1) and verification metrics (Label Accuracy and the FEVER score).

Retrieval Performance

FER demonstrated a significant leap in retrieval efficiency. It achieved an F1 score of 78.84% on the FEVER test set, representing a 23.7% improvement over the previous state-of-the-art baseline, GERE (63.74%).

Verification Accuracy

The utility of the retrieved evidence was confirmed by pairing FER with existing verifier models like GEAR and KGAT. In all cases, using FER-retrieved evidence improved the final FEVER score. For instance, the KGAT model’s FEVER score increased from 69.40 to 69.61 when it switched to evidence provided by FER.

Key Insights

Utility Over Relevance

The primary takeaway is that retrieval for a specific downstream task should be optimized for that task's success. By using "utility divergence," the retriever learns to prioritize sentences that lead the verifier to the correct conclusion, even if those sentences might not be ranked first by traditional relevance-based models.

Precision Over Recall

FER tends to retrieve more concise and precise sets of evidence. While this approach may slightly lower recall compared to models that return a larger volume of data, it significantly boosts precision. In fact-checking, providing a few highly accurate sentences is often more effective than providing a larger set that includes "noisy" or distracting information.

Model-Agnostic Feedback

The research demonstrates that FER is model-agnostic. It can be trained using feedback from various verifier architectures, adapting its retrieval behavior to suit the specific "needs" or biases of the teacher model.

Conclusion and Future Work

The shift from relevance to utility marks a significant advancement in the design of retrieval systems for fact verification. By treating the verifier as a source of feedback, FER bridges the gap between evidence extraction and claim judgment.

However, the authors note several avenues for future research. Currently, the retriever and verifier are optimized independently; a fully joint, end-to-end optimization could potentially yield even higher accuracy. Additionally, while the results on the FEVER dataset are compelling, further testing is required to see how well this feedback mechanism generalizes to other domains, such as medical or legal fact-checking. Finally, exploring more complex feedback signals—such as the direct gradients of the verification loss—could provide an even richer training signal for future retrievers.

Summary updated

BibTeX
@inproceedings{zhang2023from,
  author = {Hengran Zhang and Ruqing Zhang and Jiafeng Guo and Maarten de Rijke and Yixing Fan and Xueqi Cheng},
  title = {{From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.422},
}
Figure