TL;DR
This paper presents a robust system for retrieving relevant fact-checks for social media posts across multiple languages, tackling the challenge of misinformation spread. By combining advanced multilingual retrieval models with a strategic fine-tuning and majority voting ensemble, the system achieves high accuracy in both monolingual and crosslingual settings, demonstrating the effectiveness of these techniques for real-world fact-checking scenarios.
What Problem Does This Paper Address?
The proliferation of misinformation on social media poses a significant challenge, requiring fact-checkers to quickly and accurately verify claims, often across language barriers. SemEval-2025 Task 7 specifically addresses this by focusing on "Previously Fact-Checked Claim Retrieval" (PFCR). The task is to find the most relevant existing fact-checks for a given social media post, even when the post and fact-checks are in different languages. This is crucial because misinformation can easily cross linguistic boundaries, and manual verification for every claim in every language is infeasible. The paper aims to build a system that can effectively bridge these language gaps and identify relevant evidence for claim verification.
Methods
The authors employed a multi-stage approach:
- Preprocessing: Raw social media data was cleaned by removing noise like hashtags, emojis, and URLs. For each post, OCR outputs were combined with original text to create a comprehensive representation. This was done in both the original language and its English translation to maximize information for multilingual retrieval.
- Summarization (Exploratory): They experimented with using large language models (LLMs) like GPT-4o, Claude 3.5 Sonnet, and LLaMA-3.3-70B-Instruct to summarize posts, aiming to reduce noise and improve structure. However, this step was found to reduce accuracy, so the raw, preprocessed text was ultimately used.
- Zero-Shot Evaluation: Several state-of-the-art English and multilingual retrieval models (e.g., LaBSE, GTR-T5-Base, mE5) were evaluated in a zero-shot setting. This tested their ability to retrieve semantically similar claims without task-specific fine-tuning, using models pre-trained on large benchmarks like MTEB and MMTEB.
- Fine-tuning: The most promising models from the zero-shot evaluation were fine-tuned on the task's training data. They used a "Multiple Negatives Ranking Loss" (MNRL) to optimize the similarity between positive query-claim pairs while pushing negative pairs further apart, enhancing the model's ability to distinguish relevant from irrelevant fact-checks.
- Majority Voting Ensemble: To further improve robustness and mitigate individual model biases, an ensemble strategy was implemented. This involved combining the outputs of the best-performing models using a majority voting mechanism, where each retrieved fact-check was scored based on the model's confidence and accuracy.
Datasets
The study utilized the MultiClaim dataset, which is derived from the original MultiClaim dataset. This dataset is multilingual, covering over 150,000 fact-checks in 8 languages (English, Spanish, German, French, Arabic, Portuguese, Thai, and Malay) and social media posts in 14 languages. This dataset is central to the SemEval-2025 Task 7.
Key Insights and Ideas
- Summarization Hinders Retrieval: Surprisingly, using LLMs to summarize social media posts decreased retrieval accuracy. The authors hypothesize this is because summarization can inadvertently increase the similarity between unrelated claims, making it harder for models to differentiate them. This highlights the importance of preserving original text nuances for retrieval tasks.
- Fine-tuning is Crucial for Crosslingual Tasks: While zero-shot models show promise, fine-tuning significantly boosts performance, especially in crosslingual scenarios. This is because multilingual models, not inherently optimized for retrieval, benefit greatly from task-specific adaptation to learn cross-language mappings.
- English Models Excel on Translated Data, Multilingual Models Generalize: English-specific models performed best when applied to the English translation of the input text. However, multilingual models, while not as strong on English-only tasks, offer better generalization across a wider range of languages.
- Ensemble Methods Improve Robustness: A majority voting ensemble strategy effectively combined the strengths of multiple models, leading to more robust and accurate predictions than any single model alone.
Key Results
The paper reports impressive accuracy figures:
- The best model achieved 92% accuracy on monolingual data and 85% accuracy on crosslingual data.
- The ensemble approach consistently improved retrieval performance, showing an average improvement of 3-5 percentage points over the best single-model baseline, as measured by Success@10 (S@10).
- Figure 2 shows substantial improvements from fine-tuning across most models in both monolingual and crosslingual settings.
- Figure 3 illustrates varying performance across languages, with English showing lower accuracy due to its larger portion in the dataset, while languages with smaller fact-checking portions sometimes achieved higher accuracy due to more distinct examples.
Limitations and Future Work
The authors acknowledge several limitations:
- Struggles with Informal Language: The system tends to struggle with informal language, ambiguous expressions, and idiomatic usage, particularly in low-resource languages.
- Dataset Nuances: The fact-check dataset's nature, with many similar instances and potential lack of direct post-claim mappings, can pose challenges.
Future work suggested includes:
- Enhanced Preprocessing: Developing more sophisticated preprocessing methods to better handle informal language and ambiguity.
- Alternative Summarization: Exploring different summarization techniques that might preserve crucial information without introducing noise.
- More Language Resources: Incorporating additional language resources to improve system robustness, especially for low-resource languages.
Summary updated