From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence

Premtim Sahitaj, Jawan Kolanowski, Ariana Sahitaj, Veronika Solopova, Max Upravitelev, Daniel Röder, Iffat Maab, Junichi Yamagishi, Sebastian Möller, Vera Schmitt

TL;DR

The paper introduces PrimeFacts, a novel framework and dataset designed to unlock the rich, unstructured evidence trapped inside professional fact-checking articles. By leveraging in-text hyperlinks as natural anchors and using Large Language Models (LLMs) to decontextualize these sentences into standalone premises, the authors demonstrate massive improvements in automated evidence retrieval and claim verification.

What Problem Does This Paper Address?

Automated fact-checking (AFC) systems face a major bottleneck: a shortage of high-quality, real-world training data that pairs claims with gold-standard evidence and structured reasoning. While professional fact-checking organizations (like PolitiFact) produce thousands of articles analyzing claims, this valuable evidence remains locked in long, rhetorically complex, and unstructured text.

Simply feeding entire articles into automated systems introduces noise and reduces effectiveness. Furthermore, relying on external links often fails due to "link rot" (broken URLs). The authors tackle this challenge by finding a systematic way to isolate, extract, and clean the specific evidence-bearing sentences that journalists use to reach their verdicts.

Methods

The authors propose a three-mode extraction pipeline to transform unstructured articles into refined evidence:

  1. Mode A (Anchored Verbatim): Identifies sentences containing hyperlinks to external primary sources (e.g., government reports, datasets) and extracts them verbatim.
  2. Mode B (Decontextualization): Uses an LLM to rewrite Mode A sentences into standalone, context-independent premises. It resolves pronouns and fills in implicit context (e.g., changing "He took office then" to "Volodymyr Zelenskyy took office... in May 2019"). It also categorizes the evidence into types like Quote, Statistic, or Document.
  3. Mode C (Open Extraction): Prompts an LLM to look at the entire article and directly generate a set of self-contained premises, capturing implicit evidence that might lack an explicit hyperlink.
Figure 2: Extraction pipeline for transforming fact-checks into refined, decontextualized evidence.

To evaluate the quality of the rewrites without relying solely on standard metrics like ROUGE, the authors introduce the Decontextualization Faithfulness Score (DFS), which uses a Natural Language Inference (NLI) model to balance factual entailment against an explicit penalty for lazy, verbatim copy-pasting.

Datasets

The authors introduce the PrimeFacts dataset, compiled from 13,106 PolitiFact articles written by 661 journalists. It contains:

  • Structured article metadata (verdicts, claim statements, authors).
  • 49,718 in-article hyperlinks serving as high-quality evidence anchors.
  • LLM-generated decontextualized premises across six state-of-the-art models (including Qwen3 and Llama 3.3/4 variants).

Key Insights and Ideas

  • Hyperlinks as Natural Proxies: In-text citations are highly reliable anchors for pinpointing core evidence. Because journalists deliberately embed links to substantiate their claims, these anchors bypass the need for exhaustive, manual sentence-level annotation.
  • Portability Drives Retrievability: Stripping away discourse dependencies (like pronouns or local context) transforms the task of evidence retrieval from a complex long-context interpretation problem into a cleaner, structured factual inference problem.
  • The Open vs. Anchored Trade-off: While open extraction (Mode C) yields the highest raw performance because it aligns closely with claim wording, it only overlaps with anchor-based evidence by about 25%. This suggests a hybrid approach—using anchored links as a faithful foundation and open extraction as a supplement—is optimal.

Key Results

The framework was evaluated across two downstream tasks: cross-article evidence retrieval (using BM25) and zero-shot claim verification.

  • Evidence Retrieval: Decontextualization (Mode B) consistently outperformed verbatim text (Mode A). For example, using Llama-3.3-70B, Mode B achieved a 37% relative gain in Mean Reciprocal Rank (MRR@10) and improved Recall@10 from 0.21 to 0.36. Mode C achieved even higher retrieval scores, with MRR@10 ranging from 0.78 to 0.88 across models.
  • Claim Verification: Providing any extracted evidence dramatically beat the majority-class baseline. Moving from verbatim sentences (Mode A) to decontextualized premises (Mode B) boosted Macro-F1 scores by 4–7 points in binary classification and up to 8 points in fine-grained 5-class classification. Top-performing models (Qwen3-235B and Llama-3.3-70B) reached a binary Macro-F1 of 0.81 using Mode C evidence.

Limitations and Future Work

  • Platform Dependency: The anchor-based pipeline relies on PolitiFact’s rich hyperlinking culture. It has not yet been validated on other platforms (like Snopes or Full Fact) or non-English datasets.
  • Lack of Multi-Hop Logic: The current methodology isolates individual sentences but does not capture the overarching logical structure or argumentative chains a journalist uses to connect multiple pieces of evidence.
  • LLM Hallucinations: Despite guardrails, abstractive open extraction (Mode C) carries a minor risk of introducing altered or extraneous details. Future work will focus on linking premises into explicit argumentative graphs.

Summary updated

Abstract

Fact-checking articles encode rich supporting evidence and reasoning, yet this evidence remains largely inaccessible to automated verification systems due to unstructured presentation. We introduce PrimeFacts, a methodology and resource for extracting fine-grained evidence from full fact-checking articles. We compile 13,106 PolitiFact articles with claims, verdicts, and all referenced sources, and we identify 49,718 in-article hyperlinks as natural anchors to pinpoint key evidence. Our framework leverages large language models (LLMs) to rewrite these anchor sentences into stand-alone, context-independent premises and investigates the extraction of additional implicit evidence. In evaluations on cross-article evidence retrieval and claim verification, the extracted premises substantially improve performance. Decontextualized evidence yields higher retrievability, achieving up to a 30 percent relative gain in Mean Reciprocal Rank over verbatim sentences, and using the evidence for verdict prediction raises Macro-F1 by 10-20 points over the baseline. These gains are consistent across different verdict granularities (2-class vs. 5-class) and model architectures. A qualitative analysis indicates that the decontextualized premises remain faithful to the original sources. Our work highlights the promise of reusing fact-checkers' evidence for automation and provides a large-scale resource of structured evidence from real-world fact-checks.
BibTeX
@inproceedings{sahitaj2026from,
  author = {Premtim Sahitaj and Jawan Kolanowski and Ariana Sahitaj and Veronika Solopova and Max Upravitelev and Daniel Röder and Iffat Maab and Junichi Yamagishi and Sebastian Möller and Vera Schmitt},
  title = {{From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence}},
  booktitle = {LREC},
  year = {2026},
  doi = {10.63317/453datkp6z9s},
}
Figure