TL;DR
This paper presents UMUTeam's approach to the SemEval-2025 Task 7, focusing on retrieving previously fact-checked claims relevant to social media posts across multiple languages. They leverage XLM-RoBERTa with a metric learning strategy, including hard negative mining and Multi-Similarity Loss, to create robust cross-lingual representations, achieving over 50% hit rate across languages despite a 25th place ranking.
What Problem Does This Paper Address?
The rapid spread of misinformation on social media poses a significant threat to public discourse and societal well-being. Manually fact-checking this deluge of information is time-consuming and challenging, especially in multilingual contexts. SemEval-2025 Task 7 aims to automate a crucial part of this process: identifying relevant, pre-existing fact-checked claims for a given social media post. This task is vital for building scalable misinformation detection systems that can operate effectively across different languages.
Methods
UMUTeam's approach centers on a multilingual Transformer model, XLM-RoBERTa, to generate contextual embeddings for both social media posts and fact-checked claims. The core innovation lies in their metric learning pipeline, designed to optimize these embeddings for effective cross-lingual retrieval. Key techniques include:
- Metric Learning: Training the model to learn a shared embedding space where similar claims and posts are close together.
- Hard Negative Mining: A strategy to identify and use challenging examples (pairs that are semantically similar but not actual matches) during training. This forces the model to learn finer distinctions and improve its ability to differentiate between closely related but incorrect claims.
- Multi-Similarity Loss: A loss function that refines the similarity and dissimilarity relationships between embeddings, further enhancing the model's discriminative power.
- Single Multilingual Encoder: The system uses a single XLM-RoBERTa model to process all languages, offering a scalable and efficient solution without requiring language-specific fine-tuning.
Datasets
The authors utilized datasets provided by the SemEval-2025 organizers for both training and testing.
- Training Data: Included 153,742 facts, 24,431 posts, and 25,742 pairings linking posts to relevant fact-checked claims.
- Testing Data: Consisted of a larger set of 272,446 facts and 8,275 posts, for which the system needed to retrieve the top 10 most relevant fact-checked claims.
The paper notes that the facts set for testing was considerably large, highlighting the efficiency gains from their retrieval approach.
Key Insights and Ideas
- The Power of Multilingual Embeddings: XLM-RoBERTa's ability to generate meaningful embeddings across languages is a strong foundation for cross-lingual tasks like fact-checking retrieval.
- Hard Negatives are Crucial for Discrimination: The inclusion of hard negative mining significantly improves the model's ability to distinguish between subtly different but related claims, a common challenge in natural language understanding.
- Unified Model Efficiency: Using a single multilingual model for all languages is a practical and scalable approach, avoiding the complexity and resource demands of managing multiple language-specific models.
- Language Diversity Matters: The performance variations across languages (e.g., better performance in Thai/Malay vs. English/Polish) underscore the inherent challenges of cross-lingual NLP and the need for models that can adapt to diverse linguistic structures.
Key Results
UMUTeam achieved a 25th place ranking in the SemEval-2025 Task 7, with an average S@10 (hit rate at 10 retrieved items) of 0.544. While this performance was modest compared to the top teams (who achieved averages around 0.96), the model demonstrated consistent results across languages, achieving a hit rate above 50% for most.
- Overall Performance: Average S@10 of 0.544.
- Language Performance: Notably higher scores in Thai (0.7869) and Malay (0.6882), and lower scores in English (0.414) and Polish (0.464).
- Consistency: The model maintained a hit rate above 50% for most languages, indicating robustness.
- Efficiency: The system offers a relatively short search time due to its retrieval-based approach, making it efficient and scalable.
Limitations and Future Work
The authors acknowledge several limitations:
- No Validation Set: The model was trained on the entire training set without a separate validation split. This limited their ability to perform detailed hyperparameter tuning and ablation studies.
- Performance Gaps: While the model showed consistency, its overall performance lagged behind top-ranking systems, particularly in languages like English and Polish.
Future work directions include:
- Domain Adaptation: Exploring techniques to adapt the model to specific domains, such as political discourse or emotionally charged narratives.
- Cross-Lingual Transfer Learning: Investigating further cross-lingual transfer learning techniques.
- Addressing Complex Grammars: Developing strategies to better handle languages with complex grammatical structures.
- Joint Modeling: Exploring joint modeling approaches that integrate claim verification with stance and intention detection for more comprehensive analysis.
Summary updated