TL;DR
This paper presents the YNU-HPCC team's approach to the SemEval-2025 Task 7, focusing on multilingual and cross-lingual fact-checked claim retrieval. They developed a unified framework that integrates multiple pre-trained embedding models, fine-tuned them, and used a weighted ensemble to achieve strong performance, particularly in the challenging cross-lingual setting.
What Problem Does This Paper Address?
The paper tackles the critical challenge of automated fact-checked claim retrieval in an increasingly complex and multilingual information landscape. Manual fact-checking is time-consuming and labor-intensive, especially when claims and their corresponding evidence span multiple languages. While automated fact-checking shows promise, efficiently retrieving relevant, multilingual evidence remains a significant hurdle. This task aims to bridge this gap by developing systems that can accurately retrieve fact-checks for claims, regardless of the language involved.
Methods
The YNU-HPCC team employed a dual-encoder architecture based on the Sentence-Transformers framework. They selected four high-performing multilingual embedding models: BGE-M3, GTE-multilingual-base, Jina-embeddings-v3, and Multilingual-E5-large-instruct. These models were individually fine-tuned on the task's training data using the Hugging Face Trainer. To combine the strengths of these models, they implemented a weighted cosine similarity approach. This means that the cosine similarity scores from each individual model were combined with assigned weights to produce a final similarity score, which was then used to rank and retrieve the most relevant fact-checks.
Datasets
The experiments were conducted using datasets provided for SemEval-2025 Task 7. These datasets consist of three main components:
- Fact-checks: Contain verification results for claims made online.
- Posts: Include information about social media posts and their authenticity assessments.
- Pairs: Annotate highly relevant fact-check-post pairs, crucial for training and fine-tuning the models.
The authors also describe a preprocessing step where they filtered and augmented these datasets by extracting key information (claims, titles, OCR, verdicts) and concatenating them to create a more usable training set.
Key Insights and Ideas
- Ensemble Power: Integrating multiple embedding models significantly boosts retrieval accuracy, especially in cross-lingual scenarios. This suggests that different models capture complementary linguistic nuances.
- Translation Pitfalls: Surprisingly, translating text into English before embedding can degrade the performance of cross-lingual models. This highlights the importance of models that can directly handle multilingual inputs without relying on translation.
- Balanced Performance: The combination of Jina-embeddings-v3 and GTE-multilingual-base offered a near-optimal balance between efficiency and computational cost, achieving strong results without needing all four models.
Key Results
The YNU-HPCC system achieved competitive results in SemEval-2025 Task 7:
- Monolingual Task: An success@10 of 0.92, ranking 11th.
- Cross-lingual Task: An success@10 of 0.77, also ranking 11th.
The paper also details experimental findings on the development dataset, showing that fine-tuning for 3-4 rounds significantly improved performance. They also observed that setting max_len to 512 provided a slight improvement over 384. The ensemble of models consistently outperformed individual models, with the cross-lingual task showing more substantial gains from ensembling.
Limitations and Future Work
The authors acknowledge that their system achieved near-optimal performance with the Jina and GTE model combination, suggesting that further improvements might be marginal with their current approach.
For future work, they plan to:
- Explore the impact of dynamic weighting for the ensemble, rather than fixed weights.
- Investigate alternative ensemble methods to potentially achieve even better results.
- Devise more efficient and accurate retrieval methods using embedding models to enhance scalability for large-scale fact-checking.
Summary updated