MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters

Rrubaa Panchendrarajan, Rubén Míguez Pérez, Arkaitz ZubiagaEMNLP 2025

TL;DR

This paper introduces MultiClaimNet, a novel collection of multilingual datasets designed to group fact-checked claims that discuss the same underlying facts. By automating the creation of these claim clusters, the work aims to significantly improve the efficiency and scalability of automated fact-checking systems, particularly in handling the global spread of misinformation.

What Problem Does This Paper Address?

Automated fact-checking systems are crucial for combating the deluge of unverified information online. A key challenge is efficiently retrieving relevant evidence for claims, especially when the same claims appear across different platforms and in multiple languages. While grouping similar claims into "clusters" is a promising solution to reduce redundancy and improve retrieval, research in this area has been severely hampered by the lack of suitable, large-scale, multilingual datasets. Existing efforts have been limited in scope, language coverage, or relied on manual annotation, which is time-consuming and impractical for the scale of misinformation.

Methods

The authors propose a novel, largely automated methodology for constructing multilingual claim cluster datasets. The core idea is to leverage existing claim-matching datasets and large language models (LLMs) to identify similar claim pairs and then group them into clusters.

For the larger dataset, MultiClaim, they employ a two-step process:

  1. Sub-Cluster Creation: They first use Approximate Nearest Neighbor (ANN) search (specifically HNSW) on sentence embeddings of translated claims to identify candidate similar claim pairs. These pairs are then annotated for similarity by multiple LLMs (Falcon 11B, Falcon 40B, GPT-4, Llama3 8B, Mistral 7B, Mixtral 8×7B, and Phi3 14B). A consensus-based approach, requiring unanimous agreement from the top-performing LLMs (GPT-4, Phi-3 14B, and Falcon 40B), is used to ensure label reliability.
  2. Cluster Merging: To address potential issues from retrieving only a single nearest neighbor, they merge the initial sub-clusters. This involves computing cluster embeddings, retrieving nearest neighbors for these embeddings, and then using LLMs again to annotate inter-cluster similarity, followed by merging based on consensus.

Datasets

The paper introduces MultiClaimNet, a collection of three multilingual claim cluster datasets:

  • ClaimCheck: Derived from an existing claim-matching dataset, containing 5.2K similar claim pairs across 22 languages.
  • ClaimMatch: Also derived from an existing claim-matching dataset, with 1.5K similar claim pairs across 36 languages.
  • MultiClaim: The largest dataset, automatically constructed from the MultiClaim dataset (Pikuliak et al., 2023). It comprises 85.3K fact-checked claims written in 78 languages, forming 30.9K clusters.

Key Insights and Ideas

  • LLMs are powerful for claim similarity: The study demonstrates that LLMs, even smaller ones, can effectively identify semantically similar claims, reducing the need for extensive manual annotation.
  • Multilingualism is crucial: The datasets and experiments highlight the significant challenge and importance of handling claims across diverse languages, as similar facts are often discussed using different linguistic expressions.
  • Smaller embedding models can be sufficient: Surprisingly, the research found that smaller sentence embedding models (<1B parameters) often outperformed larger ones in clustering tasks, suggesting that efficiency and scalability are achievable.
  • Clustering algorithms have varying strengths: Different clustering algorithms perform differently across datasets of varying sizes and complexities. Agglomerative clustering showed promise for large datasets, while HDBSCAN was more effective on smaller ones.
  • Linguistic diversity is a major hurdle: The error analysis revealed that baseline approaches struggle to cluster claims that are semantically similar but expressed in different languages, underscoring the need for more robust multilingual representations.

Key Results

  • MultiClaimNet provides three multilingual claim cluster datasets, with the largest containing 85.3K fact-checked claims in 78 languages.
  • The automated methodology for constructing the MultiClaim dataset achieved a high agreement rate (77-80%) among top LLMs for claim similarity annotation.
  • Experiments showed that Agglomerative clustering performed consistently well on the largest dataset (MultiClaim), while HDBSCAN was effective on smaller datasets (ClaimCheck, ClaimMatch).
  • Smaller multilingual sentence embedding models (e.g., gte-multilingual-base, Bge-m3) achieved competitive or superior performance compared to larger models in clustering tasks.
  • The study identified that clustering algorithms are sensitive to sentence representation, and dimensionality reduction techniques (like UMAP) are beneficial.

Limitations and Future Work

  • Data Curation Bias: The smaller datasets (ClaimCheck, ClaimMatch) may exhibit biases due to their curation methods, leading to highly similar claim pairs and a limited topic range.
  • Textual Content Reliance: LLM annotations were based solely on textual content. This might lead to false positives if claims with similar text refer to different real-world incidents (e.g., multiple distinct events within a war).
  • Claims with Multiple Factual Statements: The assumption that a fact-checked claim discusses a single factual statement might lead to mismerges if a claim contains multiple distinct factual assertions.

Future work could focus on developing more robust multilingual models that can better handle linguistic diversity, exploring task-specific clustering solutions, and investigating the impact of multimodal information (beyond text) on claim clustering. The authors also suggest exploring iterative clustering for real-world integration and using claim clusters for more efficient verification of unverified claims.

Summary updated

BibTeX
@inproceedings{panchendrarajan2025multiclaimnet,
  author = {Rrubaa Panchendrarajan and Rubén Míguez Pérez and Arkaitz Zubiaga},
  title = {{MultiClaimNet: A Massively Multilingual Dataset of Fact-Checked Claim Clusters}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure