Winning ClimateCheck: A Multi-Stage System with BM25, BGE-Reranker Ensembles, and LLM-based Analysis for Scientific Abstract Retrieval

Junjun Wang, Kunlong Chen, Zhaoqun Chen, Peng He, Wenlu ZhengACL 2025

TL;DR

This paper presents ClimateCheck, a winning system for identifying relevant scientific literature to support or refute social media claims about climate change. It employs a multi-stage retrieval pipeline combining BM25 for initial candidate generation, an ensemble of fine-tuned BGE-Reranker models for precise reranking, and Gemini 2.5 Pro for claim-abstract relationship classification.

What Problem Does This Paper Address?

The paper addresses the critical challenge of combating misinformation on social media, particularly concerning climate change. The core problem is bridging the gap between the widespread dissemination of social media claims and the vast body of scientific literature that could verify or refute them. The ClimateCheck shared task aims to develop systems that can automatically retrieve relevant scientific abstracts for social media claims and then classify the relationship between the claim and the retrieved abstracts. This is crucial for informed public discourse and for enabling fact-checking efforts.

Methods

The ClimateCheck system utilizes a two-pronged approach for its two subtasks:

Subtask I: Abstract Retrieval This subtask involves a multi-stage pipeline designed to maximize both recall and precision:

  1. Initial Candidate Generation (BM25): A large corpus of approximately 400,000 scientific abstracts is first indexed using BM25. For each social media claim, BM25 is used to retrieve the top 5,000 most relevant abstracts. This stage prioritizes recall, ensuring that potentially relevant documents are not missed.
  2. Fine-Grained Reranking (BGE-Reranker Ensembles): The top 5,000 candidates from BM25 are then passed to an ensemble of fine-tuned BGE-Reranker cross-encoder models. These models excel at deep token-level interactions between claims and abstracts, providing more nuanced relevance judgments than simpler bi-encoders.
  3. Training Data: The BGE-Reranker models were trained using triplets of (claim, relevant abstract, irrelevant abstract). Crucially, the training data incorporated "hard negatives"—irrelevant abstracts that were difficult to distinguish from relevant ones—generated through a two-stage BM25 and BGE re-ranking process. This hard negative mining significantly improved the rerankers' performance.
  4. Ensembling: Multiple BGE-Reranker models were fine-tuned with varying hyperparameters. Their outputs were then ensembled using a variation of Reciprocal Rank Fusion (RRF) to combine their strengths and improve robustness.
  5. Final List Selection: The RRF-ensembled scores are used to produce the final ranked list of top-10 abstracts.

Subtask II: Claim Verification This subtask involves classifying the relationship between a claim and its retrieved abstracts into one of three categories: 'Supports', 'Refutes', or 'Not Enough Information' (NEI).

  • LLM-based Classification (Gemini 2.5 Pro): The authors leveraged the Gemini 2.5 Pro large language model for this task. A carefully crafted prompt was designed to instruct the LLM to act as a "climate change expert" and analyze the claim-abstract pairs.
  • Prompt Design: The prompt specified the task, defined the three relationship categories, and constrained the output to a single numerical digit (1, 2, or 3) for each pair. A key instruction was to aim for at least 30% 'Not Enough Information' responses to encourage conservative classifications on ambiguous cases.

Datasets

  • ClimateCheck Dataset: This dataset, provided for the shared task, was used for both training and evaluation. It contains social media claims and corresponding scholarly abstracts. The paper mentions it was used for training the reranker models and for evaluating the claim-abstract relationship classification.
  • Corpus of ~400,000 abstracts: This large corpus was used as the source for abstract retrieval.

Key Insights and Ideas

  • The Power of Multi-Stage Retrieval: The success of the system highlights the effectiveness of a cascaded approach. BM25 provides broad recall, while fine-tuned BGE-rerankers offer high precision, and ensembling further boosts robustness.
  • Importance of Hard Negative Mining: Training reranker models with carefully selected "hard negatives" proved crucial for improving their ability to distinguish between subtly different abstracts and make more accurate relevance judgments.
  • LLMs for Nuanced Classification: Gemini 2.5 Pro demonstrated strong capabilities in understanding the complex relationship between social media claims and scientific abstracts, particularly when guided by a well-designed prompt that encouraged conservative outputs.
  • Balancing Recall and Precision: The system's design explicitly balances the need for broad retrieval (recall) in the initial stage with the need for highly accurate ranking (precision) in subsequent stages.

Key Results

  • First Place in Both Subtasks: The ClimateCheck system achieved first place in both the Abstract Retrieval (Subtask I) and Claim Verification (Subtask II) of the ClimateCheck shared task.
  • Effective Reranking: The ensemble of BGE-Reranker models, trained with hard negatives, significantly improved relevance judgments over simpler methods.
  • LLM for Verification: Gemini 2.5 Pro was effectively used to classify claim-abstract relationships, providing valuable insights for fact-checking. The prompt design successfully encouraged a higher proportion of 'Not Enough Information' responses, indicating a more cautious and reliable classification.

Limitations and Future Work

  • Complexity of Ensemble Reranking: The authors acknowledge that fine-tuning and ensembling multiple reranker models, while effective, can be overly complex for real-world industrial applications.
  • LLMs in Subtask I: Despite believing LLMs could improve abstract retrieval (Subtask I), the authors were unable to practically integrate them into this subtask due to resource constraints. They suggest that future work could explore LLM-based text retrieval methods (e.g., qwen3-embedding) to potentially enhance performance without the need for complex ensembles.
  • Future Exploratory Work: The authors plan to conduct further exploratory work on LLM-based text retrieval for Subtask I.

Summary updated

BibTeX
@inproceedings{wang2025winning,
  author = {Junjun Wang and Kunlong Chen and Zhaoqun Chen and Peng He and Wenlu Zheng},
  title = {{Winning ClimateCheck: A Multi-Stage System with BM25, BGE-Reranker Ensembles, and LLM-based Analysis for Scientific Abstract Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure