Automated Fact Checking in the News Room

Sebastião Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchell, Zita Marinho

Introduction

In the modern newsroom, manual fact-checking is a daunting and tedious task. Journalists are often required to sift through massive volumes of unstructured text to verify claims under extreme time pressure. While existing tools focus on databases of previously debunked claims, there is a critical need for systems capable of handling novel, previously unchecked claims. This paper presents an end-to-end automated fact-checking platform designed to assist journalists by retrieving evidence and providing a verdict—Supports, Refutes, or Not Enough Info—for any given claim.

System Architecture

The platform utilizes a three-stage pipeline optimized for both speed and accuracy, ensuring that the entire process from claim input to verdict takes approximately 1.7 seconds.

The three-stage pipeline: Document Retrieval, Sentence Ranking, and Natural Language Inference (Hexa-F).
  1. Document Retrieval: Using the Okapi-BM25 algorithm, the system searches a database of ~445,000 news articles. By indexing both words and named entities (e.g., people, organizations), it narrows the search to roughly 5,000 relevant documents in just 50ms.
  2. Sentence Ranking: Sentences are extracted and ranked based on keyword proximity and semantic similarity (using word embeddings). The system filters for length, entity presence, and novelty to avoid redundant evidence.
  3. Natural Language Inference (NLI): The top ~25 sentences are processed by the Hexa-F model, a neural network trained on the FEVER dataset, to classify the relationship between the claim and the evidence.

Transparency and the Human-in-the-Loop

A core design principle of the system is transparency over total automation. Rather than providing a simple binary truth value, the user interface presents evidence in three distinct columns: Supports, Refutes, and Other. This layout allows journalists to act as the final arbiter, maintaining the human oversight necessary for high-stakes journalism.

The system interface, displaying evidence categorized into Supports, Refutes, and Other columns to assist journalist decision-making.

Evaluation and Key Results

The system was evaluated by 11 BBC journalists using 67 unique factual claims against a live collection of 445,000 articles. The study revealed several key performance metrics:

  • Relevance: 59% of retrieved sentences were deemed relevant to the claim.
  • Evidence Correctness: 58% of the evidence was correctly classified by the NLI model.
  • Global Correctness: The final overall verdict for a claim was correct 42% of the time.
  • The Refutation Gap: A significant finding was the difficulty in identifying refuting evidence. While precision for supporting evidence was 48%, it dropped to 27% for refutations. This is largely because supporting evidence often shares vocabulary with the claim, whereas refutations often require complex logical reasoning or different terminology.

Limitations and Future Work

While the system provides a robust framework for newsroom assistance, it faces challenges with temporal reasoning. Distinguishing between current events and similar events from the past remains difficult. Future work will focus on retrieval strategies that go beyond simple feature matching to better identify relevant but opposing arguments, as well as features that track how evidence for a claim evolves over time.

Summary updated

Abstract

Fact checking is an essential task in journalism; its importance has been highlighted due to recently increased concerns and efforts in combating misinformation. In this paper, we present an automated fact checking platform which given a claim, it retrieves relevant textual evidence from a document collection, predicts whether each piece of evidence supports or refutes the claim, and returns a final verdict. We describe the architecture of the system and the user interface, focusing on the choices made to improve its user friendliness and transparency. We conduct a user study of the fact-checking platform in a journalistic setting: we integrated it with a collection of news articles and provide an evaluation of the platform using feedback from journalists in their workflow. We found that the predictions of our platform were correct 58% of the time, and 59% of the returned evidence was relevant.
BibTeX
@inproceedings{miranda2019automated,
  author = {Sebastião Miranda and David Nogueira and Afonso Mendes and Andreas Vlachos and Andrew Secker and Rebecca Garrett and Jeff Mitchell and Zita Marinho},
  title = {{Automated Fact Checking in the News Room}},
  booktitle = {The Web Conference},
  year = {2019},
  url = {https://openreview.net/forum?id=B1ba6xW_ZH},
}
Figure