Introduction
In an era defined by information overload, statistical claims serve as the backbone of news reporting and social media discourse. However, these claims are frequently inaccurate, outdated, or misinterpreted. While manual fact-checking organizations provide a vital service, they cannot scale to meet the volume of data produced daily. Automated systems offer a potential solution, but they face a significant hurdle: numerical data in natural language is inherently "messy." Unlike structured databases, human writing often employs approximations (e.g., "nearly 2 million" instead of "2,193,843"), making it difficult for traditional algorithms to recognize and verify claims.
This paper presents a distantly supervised system designed to address the dual tasks of claim identification—locating a statistical statement within a text—and verification—checking that statement against a trusted knowledge base.
Methodology
The researchers developed a baseline approach that avoids the need for expensive, manually annotated datasets by utilizing distant supervision with Freebase as a ground-truth knowledge base. The process involves four key stages:
- Pattern Discovery: The system identifies sentences containing both a known entity (e.g., a country) and a numerical value, extracting the textual patterns that connect them.
- Greedy Pattern Selection: To filter out ambiguous or coincidental patterns, the algorithm ranks them based on their ability to predict values in the knowledge base using Mean Absolute Percentage Error (MAPE). Patterns are added to a "trusted" set only if they improve overall accuracy.
- Adjusted MAPE: A smoothing formula was introduced to penalize low-frequency patterns, ensuring the model favors phrasings that are both accurate and common.
- Verification: Identified claims are compared against the knowledge base to determine their error margin.
A core innovation of this approach is treating numerical matching as a regression problem rather than a strict string-matching task. By allowing for approximate matches during training, the system learns to recognize that phrases like "estimated at" or "has _ inhabitants" are valid indicators of a statistical property, even when the numbers are rounded.
Results and Performance
The system was evaluated against 16 statistical properties for approximately 175 countries, using a text corpus generated via the Bing Search API. The evaluation set consisted of 7,092 manually labeled statements.
The system achieved an overall precision of 60%. When used for data extraction, it reached a Mean Absolute Percentage Error (MAPE) of 0.11, indicating that its extractions were, on average, within 11% of the ground truth. However, performance varied significantly by property. While "renewable freshwater" achieved a precision of 1.00, "internet users %" fell to 0.00, largely because the patterns for the latter were frequently confused with unrelated country-ranking lists.
Discussion and Limitations
Despite its successes, the model remains limited by "context blindness." Because it focuses primarily on the patterns between an entity and a value, it can be misled by complex sentence structures or cases where the property is mentioned elsewhere in the text. Furthermore, the system is currently restricted to "Entity-Property-Value" triples and cannot yet process complex claims involving temporal changes (e.g., "inflation has risen by 2% since last year").
Future work will explore the use of continuous representations, such as word embeddings, to better capture the semantic similarity between different textual patterns and improve generalization across diverse phrasings.
Summary updated