TL;DR
This paper presents a zero-shot system for retrieving fact-checked claims across multiple languages, achieving strong results in the SemEval-2025 task. By leveraging state-of-the-art language models and a clever combination strategy, the system effectively tackles the challenge of misinformation by efficiently linking social media posts to relevant, pre-verified claims.
What Problem Does This Paper Address?
The proliferation of misinformation online, especially across different languages, poses a significant challenge to combating its spread. Manually verifying claims in various languages is time-consuming and impractical. This paper addresses the need for automated systems that can efficiently identify previously fact-checked claims relevant to social media posts, even when the posts and claims are in different languages (multilingual and cross-lingual settings). The goal is to provide a valuable tool for fact-checkers and researchers by quickly retrieving pertinent information to debunk false narratives.
Methods
The core of the proposed system is a zero-shot approach using Text Embedding Models (TEMs). This means the models are used directly without any task-specific fine-tuning on the training data. The authors experimented with several powerful TEMs, including NVIDIA NV-Embed-v2, GPT text-embedding-3-large, and Mistral mistral-embed.
Key methodological aspects include:
- Input Processing: For each social media post, the system concatenates the original text with any OCR-extracted text (from images) to maximize context. Crucially, for most models, only English translations of the text were used, as multilingual models with original texts yielded poorer results.
- Embedding Generation: Each TEM generates a vector representation (embedding) for the input post and for each candidate fact-checked claim (title + claim text).
- Similarity Matching: Cosine similarity is used to measure the semantic similarity between the post embedding and each claim embedding.
- Model Combination: For individual languages, the authors explored combining the outputs of different TEMs. This involved selecting the top claims from multiple models and merging them, carefully handling duplicates to ensure ten unique claims were retrieved.
- Zero-Shot Assumption: The system's performance is attributed to the inherent capabilities of large pre-trained language models, which can generalize to new tasks without explicit training.
Datasets
The evaluation was performed on the development data of the SemEval-2025 shared task 7. This dataset is an enhanced version of the MultiClaim dataset (Pikuliak et al., 2023). It contains tens of thousands of multilingual social media posts, each matched with over 200,000 fact-checked claims. The dataset includes the post text, OCR output (if applicable), language identification, English translations, and a verdict for each post. The task organizers provided ground truth labels for the development set.
Key Insights and Ideas
- English Translations Suffice: Surprisingly, using only English translations as input for most TEMs yielded better results than using multilingual models with original texts. This suggests that the English-centric models, despite the language barrier, captured semantic nuances effectively enough for this task.
- Larger Models Dominate: The experiments clearly showed that larger TEMs (like NV-Embed with ~7B parameters) significantly outperformed smaller models (like GTE with ~350-434M parameters), highlighting the benefit of scale in capturing complex semantic relationships.
- Model Combinations Boost Performance: For individual languages, combining different TEMs proved beneficial, leading to improved results compared to using a single model. This strategy helps leverage the strengths of diverse models.
- Zero-Shot Effectiveness: The success of the zero-shot approach demonstrates the power of modern large language models in adapting to new, complex tasks without requiring extensive task-specific training data.
Key Results
The UWBa system achieved 7th place out of 28 participants in the monolingual subtask and 9th place out of 29 participants in the cross-lingual subtask of SemEval-2025 Task 7.
- The NV-Embed-v2 model was identified as the best performing single model.
- For individual languages, combinations like GPT & NV-Embed and Mistral & NV-Embed also showed strong performance.
- In the cross-lingual scenario, the system achieved an average S@10 score of 0.783.
- Error analysis indicated that the models effectively ranked the correct fact-checks within the top positions, with around 50% of fact-checks being retrieved within the top 3 ranks.
Limitations and Future Work
- Model Limits: The system encountered limitations with maximum input sizes for some models, requiring truncation of input text.
- Cross-lingual NV-Embed: For the cross-lingual scenario, only the NV-Embed model was used, as combinations did not yield improvements. This suggests further exploration into cross-lingual model combinations might be beneficial.
- GTE/mGTE Underperformance: Smaller models like GTE and mGTE performed significantly worse, especially in cross-lingual settings, likely due to their smaller parameter sizes. While they offer faster inference, their accuracy was insufficient for this task.
- Future Work: The paper implicitly suggests that further research into optimizing model combinations and exploring more advanced cross-lingual embedding strategies could lead to even better performance. The success of NV-Embed also points towards its continued relevance for such tasks.
Summary updated