TL;DR
This paper presents a robust approach for multilingual and crosslingual fact-checked claim retrieval, achieving competitive results in SemEval-2025 Task 7. The core innovation lies in a novel weighted majority vote strategy that combines multiple fine-tuned text embedding models, significantly boosting retrieval accuracy.
What Problem Does This Paper Address?
The proliferation of misinformation on social media necessitates effective automated fact-checking systems. While significant progress has been made in monolingual fact-checking (where the claim and the post are in the same language), crosslingual fact-checking remains a significant challenge. This task involves retrieving relevant fact-checked claims for a social media post when they are in different languages. The SemEval-2025 Task 7, building upon the MultiClaim dataset, aims to push the boundaries of this crosslingual capability.
Methods
The authors employ a multi-stage training methodology for text embedding models. Initially, they leverage contrastive pre-training with in-batch negatives to learn general representations. This is followed by a crucial step of hard negatives mining, where fact-checks with high similarity scores to posts (but are not true matches) are identified and used for fine-tuning. This process helps the model learn to distinguish subtle differences.
A key contribution is their weighted majority vote strategy. Instead of simply averaging the outputs of different models, they develop an algorithm to find optimal weights for combining the predictions of multiple models (including monolingual and crosslingual variants). This is done by searching for weight configurations that maximize performance on a validation set, allowing for a more nuanced and effective aggregation of retrieval results.
Datasets
The primary dataset used is the SemEval-2025 Task 7 dataset, an enhancement of the MultiClaim dataset. This dataset comprises 280K unique fact-checks and 33K unique social media posts, spanning 8 languages in the training set (French, Spanish, English, Portuguese, Thai, German, Malay, and Arabic) and introducing Turkish and Polish in the test set. A custom validation dataset was created by splitting the training pairs for hyperparameter tuning and weight optimization.
Key Insights and Ideas
- Hard negatives are crucial but tricky: While hard negatives significantly improve fine-tuning, simply using the top few can introduce false negatives. The paper highlights the importance of carefully selecting hard negatives (e.g., the 8th to 10th ranked) to avoid degrading performance.
- Weighted majority vote outperforms naive aggregation: The proposed weighted majority vote strategy, which learns optimal weights for combining models, proved more effective than simple averaging or relying on a single best-performing model. This suggests that even models with slightly lower individual accuracies can contribute valuable information when weighted appropriately.
- Crosslingual challenges persist: The drop in accuracy on the test set for crosslingual tasks, particularly for languages not present in the training data, underscores the inherent difficulty of crosslingual retrieval and the impact of data distribution shifts.
Key Results
The UniBuc-AE team achieved competitive rankings in SemEval-2025 Task 7:
- 6th place in the monolingual track with an average S@10 score of 0.934.
- 8th place in the crosslingual track with an S@10 score of 0.79.
Fine-tuning with hard negatives led to a significant improvement of 3.5% for monolingual and 6% for crosslingual tasks. The weighted majority vote further improved results by an additional 3%.
Limitations and Future Work
The authors acknowledge that the performance drop on new languages in the test set is a limitation, likely due to distribution differences. They propose future work to explore this methodology with state-of-the-art English text embedding models like NV-Embed-v2 and E5-mistral-7b-instruct. Additionally, they suggest investigating knowledge distillation techniques to improve the performance of smaller models.
Summary updated