TL;DR
This paper investigates how to best retrieve previously fact-checked claims, especially when dealing with multiple languages. They found that while LLM-based re-ranking is the top performer in unsupervised settings, fine-tuning models with carefully selected "hard" negative examples (semantically similar but incorrect claims) significantly boosts performance, particularly in challenging crosslingual scenarios.
What Problem Does This Paper Address?
Professional fact-checkers spend a lot of time verifying information. A crucial part of their workflow is finding existing fact-checks for similar claims to avoid redundant work. This is known as Previously Fact-Checked Claim Retrieval (PFCR). While most previous work focused on English-only scenarios, the reality is that misinformation and fact-checking efforts are global. This paper tackles the challenge of making PFCR effective in multilingual (input and retrieved claims in the same language, but potentially multiple languages overall) and, more importantly, crosslingual (input and retrieved claims in different languages) settings. The core problem is how to build systems that can accurately retrieve relevant fact-checks across language barriers, which is significantly harder than within a single language.
Methods
The authors explore both unsupervised and supervised approaches for PFCR.
- Unsupervised: This involves using pre-trained text embedding models for initial retrieval, followed by a re-ranking step to improve accuracy. They compare various state-of-the-art multilingual embedding models and evaluate two re-ranking strategies: cross-encoder re-rankers and LLM-based re-rankers.
- Supervised: This approach involves fine-tuning text embedding models. The key challenge here is selecting effective "negative examples" – claims that are not relevant to the input post but might be confused with it. They propose and compare three strategies for selecting these challenging negative examples:
- Random: Picking claims randomly.
- Topic: Selecting claims from the same topic cluster as the input post.
- Similarity: Selecting claims that are semantically similar to the input post (e.g., sharing keywords or concepts). This "similarity" strategy proved to be the most effective.
Datasets
The experiments were conducted on a curated subset of the MultiClaim v2 dataset. This dataset is notable for its multilingual nature, containing posts and fact-checked claims in 47 languages, supporting 283 language combinations. The authors emphasize their careful data splitting to prevent contamination and ensure a more reliable evaluation.
Key Insights and Ideas
- Crosslingual is Harder: The crosslingual PFCR setting presents unique challenges and generally yields lower performance compared to multilingual settings, even when using the same models.
- LLMs Shine in Unsupervised Re-ranking: LLM-based re-rankers significantly outperform cross-encoder re-rankers in unsupervised PFCR, especially in crosslingual scenarios, though at a higher computational cost.
- "Hard" Negatives are Crucial for Supervised Learning: Simply using random negative examples for fine-tuning leads to rapid model saturation. Selecting semantically similar but incorrect claims ("hard negatives") is vital for effective supervised PFCR, leading to substantial performance gains.
- Similarity-Based Negative Sampling Dominates: The strategy of sampling negative examples based on semantic similarity consistently outperformed random and topic-based sampling in supervised fine-tuning.
- Model Size Isn't Everything: The study found no strong correlation between embedding model size (parameters or dimension) and performance, suggesting that factors like data coverage and pre-training methodology are more critical.
- Translation Can Still Help (Sometimes): While the paper focuses on using original languages, they found that translating to English can still be beneficial in crosslingual settings where the input and target claims differ in language, though it underperforms using original languages when they match.
Key Results
- In unsupervised settings, LLM-based re-ranking achieved the best results, with MRR@10 scores of 0.8042 (multilingual) and 0.6140 (crosslingual).
- Supervised fine-tuning using the similarity-based negative sampling strategy with multilingual-e5-large achieved significant improvements: a crosslingual MRR@10 score of 0.4947 (+6.24 points) and S@10 of 0.7076 (+7.52 points) compared to the unsupervised approach.
- The similarity sampling strategy consistently outperformed random and topic strategies across both multilingual and crosslingual settings in supervised fine-tuning.
- Cross-encoder re-ranking showed nuanced effects in multilingual settings (sometimes even hurting performance for top models) but was more consistently beneficial in crosslingual settings.
Limitations and Future Work
- Dataset Bias: The dataset might have biases, such as a disproportionate number of English posts paired with claims in other languages, and potential inaccuracies in language detection.
- Language Representation: The uneven representation of languages in the dataset could affect findings.
- Future Work: The authors suggest further investigation into the specific characteristics of the crosslingual context and exploring more advanced methods for negative example selection and re-ranking. They also highlight the potential for this work to guide the development of PFCR systems tailored to fact-checkers' needs and available computational resources.
Summary updated