Introduction
In the modern newsroom, manual fact-checking is a daunting and tedious task. Journalists are often required to sift through massive volumes of unstructured text to verify claims under extreme time pressure. While existing tools focus on databases of previously debunked claims, there is a critical need for systems capable of handling novel, previously unchecked claims. This paper presents an end-to-end automated fact-checking platform designed to assist journalists by retrieving evidence and providing a verdict—Supports, Refutes, or Not Enough Info—for any given claim.
System Architecture
The platform utilizes a three-stage pipeline optimized for both speed and accuracy, ensuring that the entire process from claim input to verdict takes approximately 1.7 seconds.
- Document Retrieval: Using the Okapi-BM25 algorithm, the system searches a database of ~445,000 news articles. By indexing both words and named entities (e.g., people, organizations), it narrows the search to roughly 5,000 relevant documents in just 50ms.
- Sentence Ranking: Sentences are extracted and ranked based on keyword proximity and semantic similarity (using word embeddings). The system filters for length, entity presence, and novelty to avoid redundant evidence.
- Natural Language Inference (NLI): The top ~25 sentences are processed by the Hexa-F model, a neural network trained on the FEVER dataset, to classify the relationship between the claim and the evidence.
Transparency and the Human-in-the-Loop
A core design principle of the system is transparency over total automation. Rather than providing a simple binary truth value, the user interface presents evidence in three distinct columns: Supports, Refutes, and Other. This layout allows journalists to act as the final arbiter, maintaining the human oversight necessary for high-stakes journalism.
Evaluation and Key Results
The system was evaluated by 11 BBC journalists using 67 unique factual claims against a live collection of 445,000 articles. The study revealed several key performance metrics:
- Relevance: 59% of retrieved sentences were deemed relevant to the claim.
- Evidence Correctness: 58% of the evidence was correctly classified by the NLI model.
- Global Correctness: The final overall verdict for a claim was correct 42% of the time.
- The Refutation Gap: A significant finding was the difficulty in identifying refuting evidence. While precision for supporting evidence was 48%, it dropped to 27% for refutations. This is largely because supporting evidence often shares vocabulary with the claim, whereas refutations often require complex logical reasoning or different terminology.
Limitations and Future Work
While the system provides a robust framework for newsroom assistance, it faces challenges with temporal reasoning. Distinguishing between current events and similar events from the past remains difficult. Future work will focus on retrieval strategies that go beyond simple feature matching to better identify relevant but opposing arguments, as well as features that track how evidence for a claim evolves over time.
Summary updated