Contrastive Retrieval Heads Improve Attention-Based Re-Ranking

Linh Tran, Yulong Li, Radu Florian, Wei SunarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) are increasingly used as re-rankers in information retrieval, yet their internal attention mechanisms often contain significant noise. This paper introduces Contrastive Retrieval (CoRe) heads, a specialized subset of attention heads (less than 1% of the total) that excel at distinguishing relevant documents from hard negatives. By isolating these heads and pruning redundant model layers, the authors achieve superior re-ranking accuracy with significantly reduced computational costs.

Introduction

In modern Information Retrieval (IR) pipelines, a fast first-stage retriever (e.g., BM25) identifies candidate documents, which are then re-ranked by a Large Language Model (LLM) to ensure high precision. While effective, standard "In-Context Re-ranking" (ICR) methods aggregate attention signals from all heads across all layers, often incorporating linguistic noise or redundant patterns that degrade ranking performance.

Researchers from Rensselaer Polytechnic Institute and IBM Research propose a surgical alternative in their paper, "Contrastive Retrieval Heads Improve Attention-Based Re-Ranking." They demonstrate that focusing on a tiny fraction of "Contrastive Retrieval" heads can lead to more accurate and efficient search systems.

Methodology: Identifying CoRe Heads

The core innovation is a Contrastive Scoring Metric used to identify the most effective attention heads. Unlike previous approaches that prioritized heads with high absolute attention to correct documents, the CoRe metric selects heads that maximize the contrast between a positive document and "hard negatives" (irrelevant documents that appear semantically similar).

The researchers found that:

  1. Sparsity is Key: Using as few as 8 heads (out of thousands) often outperforms using the entire model.
  2. Layer Specialization: CoRe heads are predominantly located in the middle layers of the transformer. This suggests that early layers handle basic syntax and late layers focus on generation, leaving the middle layers to perform semantic matching.
  3. Pruning Potential: Because the retrieval signal is concentrated in the middle, the final 50% of model layers can be pruned during inference with minimal loss in accuracy.

Experimental Results

The authors evaluated their method, CoRe-R, across several benchmarks, including BEIR (15 diverse datasets), MLDR (multilingual), and MuSiQue (multi-hop reasoning).

Figure 2: CoRe-R (using only 8 heads) consistently outperforms the full-head ICR baseline and the QR-R method across Mistral, Llama, and Phi models.

As illustrated in Figure 2, CoRe-R consistently outperforms both the standard ICR baseline and previous head-selection methods like QR-R. On the BEIR benchmark, CoRe-R improved average nDCG@10 by +3.0 points for Mistral 7B and +2.2 points for Llama-3.1 8B. Notably, heads identified on a simple English dataset (NQ) generalized effectively to multilingual tasks, indicating a fundamental retrieval mechanism within the LLM.

Efficiency and Resource Optimization

A major advantage of the CoRe-R approach is its efficiency. By isolating specific heads and pruning the latter half of the model, the researchers significantly reduced the hardware requirements for high-quality re-ranking.

Figure 6: Peak GPU memory usage decreases linearly as the percentage of pruned layers increases across various model architectures.

As shown in Figure 6, layer pruning leads to a linear reduction in peak GPU memory usage. Pruning 50% of the layers resulted in a ~40% reduction in memory and a ~20% reduction in inference latency. This optimization allows for the deployment of sophisticated re-rankers on more modest hardware without sacrificing state-of-the-art performance.

Conclusion

The discovery of Contrastive Retrieval heads challenges the assumption that "more is better" in LLM-based ranking. By identifying the specific "signal" within the model's "noise," researchers can build IR systems that are not only more accurate but also significantly leaner and faster. This work paves the way for more efficient integration of LLMs into real-world search and retrieval applications.

Summary updated

BibTeX
@misc{tran2025contrastive,
  author = {Linh Tran and Yulong Li and Radu Florian and Wei Sun},
  title = {{Contrastive Retrieval Heads Improve Attention-Based Re-Ranking}},
  year = {2025},
  eprint = {2510.02219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.02219},
}

Loading related content...

Figure