Claim Check-Worthiness Detection as Positive Unlabelled Learning

Dustin Wright, Isabelle Augenstein

Identifying check-worthy claims—the task of determining which sentences in a text warrant factual verification—is the critical first step in automated fact-checking. However, defining "check-worthiness" is inherently subjective and varies across domains. Researchers from the University of Copenhagen argue that existing datasets suffer from a "labeling gap," where many factual claims are left unlabelled simply because they were not prioritized by human annotators. To address this, they propose a novel approach: treating the problem as Positive Unlabelled (PU) Learning.

The Challenge of Claim Detection

Current datasets for political speeches, Twitter rumors, and Wikipedia citations are often treated as isolated tasks. A significant issue is that "negative" labels in these datasets are often noisy; they contain many "false negatives"—claims that are checkable but were not flagged by annotators. This inconsistency makes it difficult for standard supervised models to learn a clear boundary for check-worthiness.

Examples of check-worthy (+) and non-check-worthy (-) claims across Wikipedia, Twitter, and Politics.

The Positive Unlabelled Conversion (PUC) Method

The authors introduce Positive Unlabelled Conversion (PUC), a framework designed to identify and correct for hidden check-worthy claims. In a standard PU setting, the model works with a set of confirmed positive examples and a large pool of unlabelled data containing both hidden positives and true negatives.

The PUC process involves four key stages:

  1. Initial Classifier ($f$): A BERT-based model is trained to estimate the probability that a sample is "labelled."
  2. Prior Estimation: The model estimates the true frequency of check-worthy claims within the unlabelled pool.
  3. Relabelling: PUC ranks the unlabelled (negative) samples by their probability of being positive and "converts" the most confident ones into positive labels until the dataset matches the estimated prior.
  4. Final Classifier ($g$): A final model is trained using a weighted loss function that accounts for these newly converted positives.
The Positive Unlabelled Conversion (PUC) framework, showing the transition from initial classifier $f$ to final classifier $g$ via sample weighting.

Experimental Results and Insights

The approach was evaluated across three distinct domains: Wikipedia citations, Twitter rumors (PHEME), and political speeches. The BERT + PUC model established new state-of-the-art results on Wikipedia with an F1 score of 81.8 and on Twitter with a micro-F1 of 67.0.

A key insight from the study is the utility of Wikipedia as a universal base. Wikipedia's "citation needed" task acts as a surprisingly effective proxy for general check-worthiness. Pre-training on Wikipedia significantly boosted performance on Twitter rumors, suggesting that the linguistic markers of checkable assertions are transferable.

However, PUC did not yield improvements in the political speech domain. The authors attribute this to the "subjectivity gap": political fact-checking labels often focus on "salience" or "interest" rather than pure checkability. This violates the underlying assumptions of PU learning, as the unlabelled data contains claims that are checkable but were ignored by journalists for being uninteresting.

Conclusion

By acknowledging the uncertainty in human labels, the PU learning framework provides a more robust foundation for claim detection. While subjectivity remains a bottleneck in political discourse, the success of PUC on Wikipedia and Twitter demonstrates the potential of automated label correction in building more reliable and comprehensive fact-checking systems.

Summary updated

BibTeX
@inproceedings{wright2020claim,
  author = {Dustin Wright and Isabelle Augenstein},
  title = {{Claim Check-Worthiness Detection as Positive Unlabelled Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.43},
}
Figure