Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers from Implicit Social Media Mentions via Hybrid Retrieval and Re-Ranking

Pascal Sager, Ashwini Kamaraj, Benjamin F. Grewe, Thilo StadelmannarXiv 2025 · arXiv · OpenReview

Abstract

We present the methodology and results of the Deep Retrieval team for subtask 4b of the CLEF CheckThat! 2025 competition, which focuses on retrieving relevant scientific literature for given social media posts. To address this task, we propose a hybrid retrieval pipeline that combines lexical precision, semantic generalization, and deep contextual re-ranking, enabling robust retrieval that bridges the informal-to-formal language gap. Specifically, we combine BM25-based keyword matching with a FAISS vector store using a fine-tuned INF-Retriever-v1 model for dense semantic retrieval. BM25 returns the top 30 candidates, and semantic search yields 100 candidates, which are then merged and re-ranked via a large language model (LLM)-based cross-encoder. Our approach achieves a mean reciprocal rank at 5 (MRR@5) of 76.46% on the development set and 66.43% on the hidden test set, securing the 1st position on the development leaderboard and ranking 3rd on the test leaderboard (out of 31 teams). We achieve this strong performance by running open-source models locally and without external training data.

Introduction

There is a massive linguistic and structural chasm between a tweet and a scientific abstract. Social media posts use colloquialisms, hashtags, and summaries that rarely include formal citations or standardized terminology. Conversely, scientific literature is dense, formal, and highly specific. Existing retrieval systems often struggle with this "domain mismatch." If a user tweets about a "new study on heart issues in young people," a standard search might miss a paper titled "Incidence of Myocarditis Following mRNA Vaccination in Adolescents."

To address this, the "Deep Retrieval" team from the University of Zurich and ZHAW developed a system designed to handle implicit mentions by understanding both literal lexical overlaps and underlying semantic context.

Methodology: The Hybrid Retrieval Pipeline

The proposed system consists of a three-stage hybrid retrieval and re-ranking pipeline designed to bridge the gap between informal social media language and formal scientific discourse.

Overview of the hybrid retrieval pipeline and the challenge of identifying scientific documents from implicit social media mentions.

1. Lexical Retrieval (BM25)

This stage focuses on exact term overlaps, such as named entities or specific keywords. A key innovation in this implementation was swapping standard lemmatization for Byte Pair Encoding (BPE) subword tokenization. This allows the system to find partial matches between informal and formal terms, which is crucial for handling the messy vocabulary of social media.

To capture context beyond exact keywords, the team utilized a fine-tuned version of INF-Retriever-v1 (based on the Qwen2-7B architecture). The model was fine-tuned specifically on the competition's training data using a "multiple negatives ranking" loss. This stage converts both the tweets and the papers into high-dimensional vectors, allowing for retrieval based on conceptual similarity.

3. LLM-based Re-Ranking

The top 130 candidates from the first two stages are fed into a powerful cross-encoder (bge-reranker-v2-gemma). Unlike the previous stages which look at documents in isolation, the re-ranker evaluates the tweet and the paper abstract together to score their relationship, providing a final deep contextual check.

Experimental Results

The study utilized the official CLEF CheckThat! 2025 Subtask 4b dataset, comprising a corpus of 7,718 scientific documents and nearly 3,000 queries across development and test sets. The primary metric was Mean Reciprocal Rank at 5 (MRR@5).

  • Leaderboard Performance: The system achieved an MRR@5 of 76.46% on the development set (1st place) and 66.43% on the hidden test set (3rd place out of 31 teams).
  • The Re-Ranking Boost: Adding the LLM-based re-ranker provided a massive 23.3 percentage point improvement over the baseline BM25 system.
  • Hybrid Synergy: Neither keyword search nor semantic search was sufficient alone. BM25 reached ~51% MRR@5, while the fine-tuned semantic retriever reached ~56%. Their combination, followed by re-ranking, was essential for top-tier performance.

Discussion and Future Work

Several key insights emerged from the development process. First, fine-tuning remains critical; even powerful foundation models like Qwen2 benefited from a 2-percentage-point boost through in-domain training. Second, the team found that while LLM-based query expansion (rewriting tweets into formal language) improved precision, it made the system 60 times slower, leading them to favor the more efficient re-ranking approach.

Future work will address current limitations, such as the neglect of document-level metadata (e.g., publication dates) and the reliance on a closed corpus. Integrating metadata-aware retrieval and external web search could further sharpen the accuracy of scientific source attribution in real-world scenarios.

Summary updated

Abstract

We present the methodology and results of the Deep Retrieval team for subtask 4b of the CLEF CheckThat! 2025 competition, which focuses on retrieving relevant scientific literature for given social media posts. To address this task, we propose a hybrid retrieval pipeline that combines lexical precision, semantic generalization, and deep contextual re-ranking, enabling robust retrieval that bridges the informal-to-formal language gap. Specifically, we combine BM25-based keyword matching with a FAISS vector store using a fine-tuned INF-Retriever-v1 model for dense semantic retrieval. BM25 returns the top 30 candidates, and semantic search yields 100 candidates, which are then merged and re-ranked via a large language model (LLM)-based cross-encoder. Our approach achieves a mean reciprocal rank at 5 (MRR@5) of 76.46% on the development set and 66.43% on the hidden test set, securing the 1st position on the development leaderboard and ranking 3rd on the test leaderboard (out of 31 teams), with a relative performance gap of only 2 percentage points compared to the top-ranked system. We achieve this strong performance by running open-source models locally and without external training data, highlighting the effectiveness of a carefully designed and fine-tuned retrieval pipeline.
BibTeX
@misc{sager2025deep,
  author = {Pascal Sager and Ashwini Kamaraj and Benjamin F. Grewe and Thilo Stadelmann},
  title = {{Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers from Implicit Social Media Mentions via Hybrid Retrieval and Re-Ranking}},
  year = {2025},
  eprint = {2505.23250},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.23250},
}
Figure