ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking

Xianming Li, Aamir Shakir, Rui Huang, Julius Lipp, Benjamin Clavié, Jing Li

TL;DR

ProRank is a novel two-stage training framework that enables Small Language Models (SLMs) to punch far above their weight class in document reranking. By combining reinforcement learning for "prompt warmup" with fine-grained score learning, the authors produced a 0.5B parameter model that outperforms 32B parameter giants and top-tier proprietary models on standard benchmarks.

What Problem Does This Paper Address?

In the world of Information Retrieval (IR) and Retrieval-Augmented Generation (RAG), reranking is the "refining" step. After a fast retriever (like BM25) pulls a list of candidate documents, a reranker re-orders them to ensure the most relevant ones are at the top.

While Large Language Models (LLMs) are excellent at this via zero-shot prompting, they are computationally expensive and slow. Small Language Models (SLMs) are the obvious efficient alternative, but they have a "prompt alignment" problem: without extensive fine-tuning, they struggle to understand complex task prompts and often fail to output results in the required format. This paper aims to bridge that gap, making SLMs viable, high-performance rerankers.

Methods

The authors introduce ProRank, a two-stage approach:

  1. Prompt Warmup (Reinforcement Learning): Instead of standard Supervised Fine-Tuning (SFT), the authors use Group Relative Policy Optimization (GRPO). This stage "steers" the SLM to follow the prompt and output a binary relevance score (0 or 1). They use a dual-reward system: a format reward (did the model output a valid 0 or 1?) and an accuracy reward (is the judgment correct?).
  2. Fine-grained Score Learning: Binary scores (0/1) are too coarse for ranking—you can't distinguish between ten documents all labeled "1." To fix this, ProRank aggregates the logits (raw prediction probabilities) of the "0" and "1" tokens to produce a continuous relevance score. The model is then further fine-tuned using a Cross-Encoder approach with binary cross-entropy loss to sharpen these scores.

This method is particularly clever because it improves performance without adding any new layers or architectural complexity to the model.

Datasets

The models were trained primarily on the bge-m3 dataset, a standard for multilingual retrieval. Evaluation was extensive, covering:

  • BEIR Benchmark: 14 English datasets across various domains (e.g., SciFact, NFCorpus, HotpotQA).
  • CMTEB: Chinese retrieval datasets (Covid and DuReader).
  • COIR: The COSQA dataset for code-specific reranking.

Key Insights and Ideas

  • RL over SFT: The authors found that Reinforcement Learning (GRPO) is significantly more effective than standard SFT for teaching SLMs to follow prompts. On the NFCorpus dataset, the RL-warmed model achieved 62.27% accuracy compared to just 40.54% for the SFT version.
  • The Scaling Paradox: Interestingly, "coarse-grained" models (those just outputting 0/1) actually saw performance decrease when scaling from 0.5B to 1.5B parameters. However, the "fine-grained" models (using logits) scaled predictably and positively. This suggests that fine-grained scoring is more robust and better at leveraging increased model capacity.
  • Efficiency vs. Power: The most striking insight is that a properly "warmed up" 0.5B model can outperform a 32B model. This challenges the "bigger is always better" narrative in reranking and highlights the importance of training strategy over raw parameter count.

Key Results

  • Beating the Giants: ProRank-0.5B (based on Qwen2.5) achieved an average NDCG@10 of 55.57 on the BEIR benchmark, surpassing the Rank1-32B model.
  • Competitive with Proprietary Tech: ProRank models consistently outperformed or matched high-end proprietary rerankers like Cohere-rerank-3.5 and Voyage-rerank-2.
  • Cross-Lingual Success: The model showed strong "zero-shot" transfer to Chinese and Code domains, even when primarily trained on English-heavy data, proving its generalizability.

Limitations and Future Work

  • The Noise Ceiling: The authors noted that performance tends to peak when reranking the top 100–1,000 documents. At very high top-k values (e.g., k=5,000), performance slightly degrades as the model encounters more "noise" (non-relevant documents) that can distract the scoring mechanism.
  • Future Directions: The team suggests exploring adaptive top-k selection—dynamically deciding how many documents to rerank based on the query—and further refining the architecture to be even more robust against noisy, irrelevant candidates.

Summary updated

Abstract

Reranking is fundamental to information retrieval and retrieval-augmented generation, with recent Large Language Models (LLMs) significantly advancing reranking quality. Most current works rely on large-scale LLMs (>7B parameters), presenting high computational costs. Small Language Models (SLMs) offer a promising alternative because of computational efficiency. However, our preliminary quantitative analysis reveals key limitations of SLMs: their representation space is narrow, leading to reduced expressiveness, and they struggle with understanding task prompts without fine-tuning. To address these issues, we introduce a novel two-stage training approach, ProRank, for SLM-based document reranking. We propose using reinforcement learning to improve the understanding of task prompts. Additionally, we introduce fine-grained score learning to enhance representation expressiveness and further improve document reranking quality. Extensive experiments suggest that ProRank consistently outperforms both the most advanced open-source and proprietary reranking models. Notably, our 0.5B ProRank even surpasses powerful LLM reranking models on the BEIR benchmark, establishing that properly trained SLMs can achieve superior document reranking performance while maintaining computational efficiency.
BibTeX
@inproceedings{li2026prorank,
  author = {Xianming Li and Aamir Shakir and Rui Huang and Julius Lipp and Benjamin Clavié and Jing Li},
  title = {{ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.51},
}

Loading related content...

Figure