Claim Matching Beyond English to Scale Global Fact-Checking

Ashkan Kazemi, Kiran Garimella, Devin Gaffney, Scott A. HaleACL 2021

TL;DR

This paper tackles the challenge of scaling fact-checking efforts globally by developing a method for "claim matching" – identifying different messages that convey the same factual claim. They introduce a new multilingual dataset and a novel embedding model (I-XLM-R) that significantly outperforms existing models in matching claims across languages, paving the way for more efficient and comprehensive misinformation detection.

What Problem Does This Paper Address?

Fact-checking is crucial for combating misinformation, but it's a labor-intensive process that doesn't scale well, especially in non-English languages. When a fact-check is performed on a specific claim, it's highly desirable to identify all other instances of that same claim circulating online, regardless of how it's phrased or in which language. This allows fact-checkers to prioritize content that is already widespread and to serve relevant fact-checks to users through "misinformation tiplines." The current challenge is efficiently and accurately matching these claims, particularly across the diverse linguistic landscape of the internet.

Methods

The core of this work lies in developing and evaluating methods for claim matching. The authors propose a two-pronged approach:

  1. Information Retrieval (IR) Approach: This method uses a baseline like BM25 (a standard search algorithm) and then reranks the results using embedding similarity. The idea is to first retrieve potentially relevant messages and then use sophisticated embeddings to sort them by how likely they are to contain the same claim.
  2. Classification Approach: This method directly predicts whether two messages contain matching claims. They explore two types of classifiers:
  3. Threshold Classifier: This simple classifier uses a predefined threshold on the cosine similarity of embeddings. If the similarity is above the threshold, the claims are considered a match.
  4. Claim Matching Classifier: This uses a more complex binary classifier (AdaBoost) that leverages features like message lengths, embedding vectors, and their cosine similarity to make a prediction.

A key technical contribution is the training of a multilingual embedding model called I-XLM-R. This model is trained using a knowledge distillation approach, where a high-quality English "teacher" model guides the learning of a multilingual "student" model (XLM-R). This helps to bridge the gap in embedding quality between high-resource (like English) and lower-resource languages (like Bengali, Malayalam, Tamil) present in their dataset.

Datasets

The paper introduces two novel datasets:

  1. Claim-like Statements Dataset: This dataset consists of 5,066 messages from WhatsApp tiplines and public groups across English, Hindi, Bengali, Malayalam, and Tamil. These messages were annotated to identify "claim-like statements."
  2. Claim Similarity Dataset: This dataset comprises 2,343 pairs of social media messages and fact-checks, also in the five languages. These pairs are annotated for claim similarity, indicating whether they can be served by the same fact-check.

These datasets are designed to address the limitations of previous work, which often focused solely on English or specific political contexts, and to enable claim matching across high- and low-resource languages.

Key Insights and Ideas

  • Multilingualism is Key: The paper strongly emphasizes that effective misinformation combat requires tackling claims in all languages, not just English. Their work demonstrates that existing multilingual models can be improved for this specific task.
  • Claim Matching is Harder than STS: While related to Semantic Textual Similarity (STS), claim matching has nuances. Messages might not have the exact same meaning but could still convey the same factual claim, requiring a more specialized approach.
  • Knowledge Distillation for Low-Resource Languages: The I-XLM-R model, trained via knowledge distillation, shows that it's possible to significantly improve embedding quality for lower-resource languages by leveraging knowledge from high-resource languages.
  • The Power of Embeddings: The results consistently show that embedding-based approaches, particularly their I-XLM-R model, are superior to simpler methods for claim matching, both in retrieval and classification tasks.
  • Fact-checks vs. Social Media: Matching fact-checks to other fact-checks is slightly easier than matching social media messages to each other, likely due to the more standardized and less varied language used in fact-checks.

Key Results

  • I-XLM-R Outperforms Baselines: The newly trained I-XLM-R model consistently outperformed established multilingual embedding models like LASER and LaBSE in claim matching tasks across various languages.
  • Information Retrieval: In reranking tasks (using BM25 + embeddings), I-XLM-R achieved the highest Mean Reciprocal Rank (MRR) scores in most languages, notably showing a significant lead in Bengali.
  • Classification: For binary classification of matching claims, I-XLM-R achieved the highest average F1 scores (up to 0.88) across all languages, outperforming LASER and LaBSE.
  • Multilingual Clusters: Preliminary experiments showed that I-XLM-R could successfully cluster messages from different languages that conveyed the same claims, demonstrating its potential for multilingual claim matching.

Limitations and Future Work

  • Data Imbalance: The paper notes that there were "too few examples of fact-checks correctly matched a social media item" to reliably evaluate performance in that specific setting. However, they argue this is less of a limitation in practice, as most fact-checks originate from social media.
  • English Model Performance: While I-XLM-R performed well, the English-only SBERT model showed slightly better performance on English data, suggesting potential "drift" from the teacher model during training.
  • Ambiguous Cases: Errors in classification often occurred in ambiguous cases or where context was missing, which human annotators could resolve but models struggled with.
  • Future Work: The authors suggest further work on multilingual claim matching, exploring more complex classification models, and expanding the dataset to cover more languages and domains. They also highlight the potential for their system to be integrated into real-world fact-checking workflows.

Summary updated

BibTeX
@inproceedings{kazemi2021claim,
  author = {Ashkan Kazemi and Kiran Garimella and Devin Gaffney and Scott A. Hale},
  title = {{Claim Matching Beyond English to Scale Global Fact-Checking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure