"Who said it, and Why?" Provenance for Natural Language Claims

Yi Zhang, Zachary G. Ives, Dan RothACL 2020 · DOI

Abstract

This research introduces a formal framework for claim provenance, moving beyond binary veracity labels to map how claims originate and evolve across the web. By modeling claim history as a directed graph and utilizing a novel Textual Entailment (TE) approach for source identification, the authors demonstrate that understanding a claim's lineage significantly enhances the accuracy of automated fact-checking systems.

Introduction: The Provenance Problem

In the current landscape of "information pollution," fact-checkers face an overwhelming volume of data. While existing AI tools primarily focus on determining whether a claim is "real" or "fake," they often neglect its provenance: the origin and evolution of the statement.

Traditional systems treat every mention of a claim as independent evidence. However, if multiple websites repeat the same original falsehood, they should not be treated as independent "votes" for truth. This paper addresses the challenge of reconstructing a "provenance graph"—a causal chain of statements and sources—to provide necessary context and prevent the overcounting of dependent sources, a phenomenon known as "collusion."

Methodology: Reconstructing the Provenance Graph

The authors propose a three-step pipeline to infer a claim's provenance:

  1. Claim Search: A corpus is searched for sentences related to a query claim, focusing on content published prior to the claim in question.
  2. Source Extraction via Textual Entailment (TE): This core innovation frames source identification as a TE problem rather than standard sequence tagging. The model asks: Does this context entail that [Source] said [Claim]? A BERT-based model, fine-tuned with margin ranking loss, distinguishes true sources from unrelated entities.
  3. Graph Construction: Extracted sources are linked using Wikification and ELMo-based similarity to group identical statements. Edges in the graph represent relationships such as identical, paraphrased, textually entailed, or motivated (where one claim triggers another).
A provenance graph illustrating the causal chain and evolution of a claim across different sources.

Experimental Evaluation

The framework was evaluated using two primary datasets:

  • MPQA 2.0: A corpus of 535 documents manually annotated with opinions and sources, used for training the source extraction model.
  • Fact-Check Dataset: 495 fact-check questions crawled from factcheck.org, including claims, veracity labels, and summaries, used to evaluate the graph's utility in verification.

Results and Insights

The experimental results highlight the superiority of the TE-based approach and the value of provenance in fact-checking.

Source Extraction Performance

The proposed TE-based model (TE-D) achieved an F1 score of 0.76, significantly outperforming the standard sequential tagging baseline (SEQ), which only reached an F1 of 0.39. This confirms that identifying "who said what" requires the deep semantic reasoning provided by transformer-based TE models, as relationships are often implicit.

Performance metrics for source extraction, comparing the proposed TE models against the sequential baseline.

Impact on Claim Verification

By identifying "sink nodes" (original sources) in the provenance graph, the system can weight independent evidence more heavily. The Prov-Src method, which votes based on independent source nodes, significantly outperformed baselines that merely aggregated top search results. This suggests that the structural context of a claim is as vital as its content.

Conclusion and Future Work

While the provenance framework marks a significant step forward, limitations remain. The current model struggles with implicit authorship where the source is not explicitly named. Future research may incorporate stylometry to identify authors based on writing patterns or explore the long-term trustworthiness dynamics of specific entities.

Summary updated

BibTeX
@inproceedings{zhang2020who,
  author = {Yi Zhang and Zachary G. Ives and Dan Roth},
  title = {{"Who said it, and Why?" Provenance for Natural Language Claims}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.406},
}
Figure