What is Your Article Based On? Inferring Fine-grained Provenance

Yi Zhang, Zachary G. Ives, Dan RothACL 2021 · DOI

Introduction

In an era of rampant misinformation, verifying the trustworthiness of an article requires more than just identifying its publisher. Real-world articles are complex tapestries of claims, paraphrases, and summaries built upon external information. To truly evaluate them, readers need fine-grained provenance: the ability to trace specific sentences back to their original source URLs. This paper introduces a novel framework to automate this process, moving beyond simple fact-checking of isolated claims to holistic article verification.

The Provenance Challenge

Most existing research focuses on "fact-checking" single claims. However, complex articles present unique challenges:

  1. Paraphrasing: Sentences are often rewritten, making keyword-based retrieval ineffective.
  2. Selectivity: Not every sentence requires a source; identifying "source-heavy" content is a prerequisite.
  3. Global Context: Sources are often related (e.g., multiple citations from the same domain), a fact ignored by sentence-level models.

Methodology: A Three-Stage Pipeline

The authors propose a comprehensive pipeline to infer provenance, transitioning from raw text to a global set of source candidates.

Overview of the three-stage pipeline for inferring fine-grained provenance: Sentence Ranking, Candidate Generation, and Joint Inference.

1. Sentence Ranking

The first stage identifies sentences containing external information. Using a RoBERTa-based model (SR-TE), the system leverages the article's main topic and key entities to distinguish between original author commentary and factual claims derived from outside sources.

2. Candidate Generation (BART-SR)

Instead of searching for raw sentences, the system generates "metadata" queries (likely titles, dates, or domains). The core of this stage is BART-SR, which utilizes a rank-aware multi-head cross-attention mechanism.

The BART-SR architecture incorporates search engine signals using rank-aware multi-head cross-attention to generate accurate source metadata.

By "attending" to preliminary search results and weighting higher-ranked results more heavily, the model refines its query generation, significantly improving retrieval performance over standard text-based searches.

3. Joint Inference (ILP)

The final stage uses Integer Linear Programming (ILP) to select the most probable sources for the entire article. This "joint" approach assumes authors maintain consistency—for instance, citing multiple articles from the same reputable domain. The ILP framework optimizes for a global set of sources, ensuring the final selection is contextually coherent.

The Politi-Prov Dataset

To evaluate their approach, the authors introduced Politi-Prov, a benchmark dataset scraped from PolitiFact.com. It contains 1,765 articles where the "Our Sources" section serves as the ground truth for provenance. Unlike claim-evidence datasets, Politi-Prov maps full articles to specific source URLs, providing a realistic testbed for fine-grained inference.

Key Results and Insights

The proposed system demonstrated significant performance gains:

  • Retrieval Accuracy: Searching for generated metadata (MS1) achieved a mean recall of 0.30, a massive jump from the 0.067 recall of raw sentence searches (SS1).
  • The Power of Jointness: Adding ILP inference (MS-ILP) further boosted recall to 0.37, nearing the search engine's theoretical upper bound of 0.39.
  • Metadata Superiority: The results confirm that predicting the title of a source is far more effective for retrieval than using the paraphrased sentence itself.

Conclusion and Future Directions

While the framework marks a significant step forward, a "bottleneck" remains in the query generation phase. Even with perfect metadata, search engines sometimes fail to locate specific documents. Future work will focus on enhancing text generation models and incorporating additional contextual signals, such as image captions, to further bridge the gap between an article and its origins.

Summary updated

BibTeX
@inproceedings{zhang2021what,
  author = {Yi Zhang and Zachary G. Ives and Dan Roth},
  title = {{What is Your Article Based On? Inferring Fine-grained Provenance}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.458},
}
Figure