YNU-HPCC at SemEval-2025 Task 7: Multilingual and Cross-lingual Fact-checked Claim Retrieval

Yuheng Mao, Jin Wang, Xuejie ZhangACL 2025

TL;DR

This paper presents the YNU-HPCC team's approach to the SemEval-2025 Task 7, focusing on multilingual and cross-lingual fact-checked claim retrieval. They developed a unified framework that integrates multiple pre-trained embedding models, fine-tuned them, and used a weighted ensemble to achieve strong performance, particularly in the challenging cross-lingual setting.

What Problem Does This Paper Address?

The paper tackles the critical challenge of automated fact-checked claim retrieval in an increasingly complex and multilingual information landscape. Manual fact-checking is time-consuming and labor-intensive, especially when claims and their corresponding evidence span multiple languages. While automated fact-checking shows promise, efficiently retrieving relevant, multilingual evidence remains a significant hurdle. This task aims to bridge this gap by developing systems that can accurately retrieve fact-checks for claims, regardless of the language involved.

Methods

The YNU-HPCC team employed a dual-encoder architecture based on the Sentence-Transformers framework. They selected four high-performing multilingual embedding models: BGE-M3, GTE-multilingual-base, Jina-embeddings-v3, and Multilingual-E5-large-instruct. These models were individually fine-tuned on the task's training data using the Hugging Face Trainer. To combine the strengths of these models, they implemented a weighted cosine similarity approach. This means that the cosine similarity scores from each individual model were combined with assigned weights to produce a final similarity score, which was then used to rank and retrieve the most relevant fact-checks.

Datasets

The experiments were conducted using datasets provided for SemEval-2025 Task 7. These datasets consist of three main components:

  • Fact-checks: Contain verification results for claims made online.
  • Posts: Include information about social media posts and their authenticity assessments.
  • Pairs: Annotate highly relevant fact-check-post pairs, crucial for training and fine-tuning the models.

The authors also describe a preprocessing step where they filtered and augmented these datasets by extracting key information (claims, titles, OCR, verdicts) and concatenating them to create a more usable training set.

Key Insights and Ideas

  • Ensemble Power: Integrating multiple embedding models significantly boosts retrieval accuracy, especially in cross-lingual scenarios. This suggests that different models capture complementary linguistic nuances.
  • Translation Pitfalls: Surprisingly, translating text into English before embedding can degrade the performance of cross-lingual models. This highlights the importance of models that can directly handle multilingual inputs without relying on translation.
  • Balanced Performance: The combination of Jina-embeddings-v3 and GTE-multilingual-base offered a near-optimal balance between efficiency and computational cost, achieving strong results without needing all four models.

Key Results

The YNU-HPCC system achieved competitive results in SemEval-2025 Task 7:

  • Monolingual Task: An success@10 of 0.92, ranking 11th.
  • Cross-lingual Task: An success@10 of 0.77, also ranking 11th.

The paper also details experimental findings on the development dataset, showing that fine-tuning for 3-4 rounds significantly improved performance. They also observed that setting max_len to 512 provided a slight improvement over 384. The ensemble of models consistently outperformed individual models, with the cross-lingual task showing more substantial gains from ensembling.

Limitations and Future Work

The authors acknowledge that their system achieved near-optimal performance with the Jina and GTE model combination, suggesting that further improvements might be marginal with their current approach.

For future work, they plan to:

  • Explore the impact of dynamic weighting for the ensemble, rather than fixed weights.
  • Investigate alternative ensemble methods to potentially achieve even better results.
  • Devise more efficient and accurate retrieval methods using embedding models to enhance scalability for large-scale fact-checking.

Summary updated

BibTeX
@inproceedings{mao2025ynu,
  author = {Yuheng Mao and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2025 Task 7: Multilingual and Cross-lingual Fact-checked Claim Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure