Identification and Verification of Simple Claims about Statistical Properties

Andreas Vlachos, Sebastian RiedelEMNLP 2015 · DOI

Introduction

In an era defined by information overload, statistical claims serve as the backbone of news reporting and social media discourse. However, these claims are frequently inaccurate, outdated, or misinterpreted. While manual fact-checking organizations provide a vital service, they cannot scale to meet the volume of data produced daily. Automated systems offer a potential solution, but they face a significant hurdle: numerical data in natural language is inherently "messy." Unlike structured databases, human writing often employs approximations (e.g., "nearly 2 million" instead of "2,193,843"), making it difficult for traditional algorithms to recognize and verify claims.

This paper presents a distantly supervised system designed to address the dual tasks of claim identification—locating a statistical statement within a text—and verification—checking that statement against a trusted knowledge base.

Methodology

The researchers developed a baseline approach that avoids the need for expensive, manually annotated datasets by utilizing distant supervision with Freebase as a ground-truth knowledge base. The process involves four key stages:

  1. Pattern Discovery: The system identifies sentences containing both a known entity (e.g., a country) and a numerical value, extracting the textual patterns that connect them.
  2. Greedy Pattern Selection: To filter out ambiguous or coincidental patterns, the algorithm ranks them based on their ability to predict values in the knowledge base using Mean Absolute Percentage Error (MAPE). Patterns are added to a "trusted" set only if they improve overall accuracy.
  3. Adjusted MAPE: A smoothing formula was introduced to penalize low-frequency patterns, ensuring the model favors phrasings that are both accurate and common.
  4. Verification: Identified claims are compared against the knowledge base to determine their error margin.
Examples of textual patterns and associated numerical values for the 'population' property, illustrating the variety of natural language phrasings.

A core innovation of this approach is treating numerical matching as a regression problem rather than a strict string-matching task. By allowing for approximate matches during training, the system learns to recognize that phrases like "estimated at" or "has _ inhabitants" are valid indicators of a statistical property, even when the numbers are rounded.

Results and Performance

The system was evaluated against 16 statistical properties for approximately 175 countries, using a text corpus generated via the Bing Search API. The evaluation set consisted of 7,092 manually labeled statements.

Precision and claim counts across various Freebase properties, showing an overall precision of 0.60.

The system achieved an overall precision of 60%. When used for data extraction, it reached a Mean Absolute Percentage Error (MAPE) of 0.11, indicating that its extractions were, on average, within 11% of the ground truth. However, performance varied significantly by property. While "renewable freshwater" achieved a precision of 1.00, "internet users %" fell to 0.00, largely because the patterns for the latter were frequently confused with unrelated country-ranking lists.

Discussion and Limitations

Despite its successes, the model remains limited by "context blindness." Because it focuses primarily on the patterns between an entity and a value, it can be misled by complex sentence structures or cases where the property is mentioned elsewhere in the text. Furthermore, the system is currently restricted to "Entity-Property-Value" triples and cannot yet process complex claims involving temporal changes (e.g., "inflation has risen by 2% since last year").

Future work will explore the use of continuous representations, such as word embeddings, to better capture the semantic similarity between different textual patterns and improve generalization across diverse phrasings.

Summary updated

BibTeX
@inproceedings{vlachos2015identification,
  author = {Andreas Vlachos and Sebastian Riedel},
  title = {{Identification and Verification of Simple Claims about Statistical Properties}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1312},
}
Figure