Abstract
We present a novel approach for training small language models for reasoning-intensive document ranking that combines knowledge distillation with reinforcement learning optimization. While existing methods often rely on expensive human annotations or large black-box language models, our methodology leverages web data and a teacher LLM to automatically generate high-quality training examples with relevance explanations. By framing document ranking as a reinforcement learning problem and incentivizing explicit reasoning capabilities, we train a compact 3B parameter language model that achieves state-of-the-art performance on the BRIGHT benchmark. Our model ranks third on the leaderboard while using substantially fewer parameters than other approaches, outperforming models that are over 20 times larger. Through extensive experiments, we demonstrate that generating explanations during inference, rather than directly predicting relevance scores, enables more effective reasoning with smaller language models. The self-supervised nature of our method offers a scalable and interpretable solution for modern information retrieval systems.
Introduction
Modern search engines and Retrieval-Augmented Generation (RAG) systems are increasingly tasked with handling "reasoning-intensive" queries. These queries—ranging from complex coding errors to multi-step logical deductions—require more than simple keyword or semantic matching. While Large Language Models (LLMs) like GPT-4 or Llama-70B possess the necessary reasoning capabilities, their computational overhead makes them impractical for high-throughput, real-time document re-ranking.
This research introduces InteRank, a framework designed to distill the reasoning power of massive models into a compact 3B-parameter footprint. By combining synthetic data distillation with reinforcement learning (RL), InteRank enables a small model to outperform models 20 times its size on complex retrieval tasks.
The InteRank Framework
InteRank utilizes a three-phase pipeline to transition a base Llama 3.2 3B model into a specialized reasoning-heavy re-ranker.
1. Synthetic Data Generation
To avoid the high costs of human annotation, the authors curated a dataset of 20,000 question-answer pairs from 186 StackExchange communities (covering math, science, and programming). A "teacher" model (Llama 3.3 70B) was employed to generate step-by-step explanations and relevance labels (on a scale of 0 to 2) for these pairs, creating a high-quality ground truth for reasoning.
2. Supervised Fine-Tuning (SFT)
The 3B student model undergoes initial training to mimic the teacher’s reasoning paths and final relevance scores. This phase establishes a baseline capability for the model to follow instructions and generate coherent justifications for its ranking decisions.
3. Reinforcement Learning (RL) Refinement
To move beyond simple imitation, the framework employs a variant of the REINFORCE algorithm. A separate reward model (Skywork-Reward) evaluates the quality of the student's reasoning. This phase encourages the model to explore and internalize more effective logical paths, refining its ability to distinguish between relevant and irrelevant documents based on logical merit rather than just pattern matching.
Experimental Setup and Results
The model was evaluated on the BRIGHT benchmark, a challenging dataset specifically designed for reasoning-intensive retrieval across domains like Biology, Economics, and Theorem Proving. InteRank acts as a second-stage re-ranker, processing the top 100 documents provided by an initial retriever.
Key Performance Metrics
- State-of-the-Art Efficiency: InteRank (3B) achieved an average nDCG@10 of 27.4% on BRIGHT.
- Leaderboard Success: The model secured 3rd place on the BRIGHT leaderboard, outperforming Reason-to-Rank (8B) and even matching or exceeding the performance of its own 70B teacher in zero-shot settings.
- Significant Gains: InteRank showed substantial improvements over traditional cross-encoders (like MiniLM) and dense retrievers (like BGE or E5), which typically scored below 20% on the same tasks.
Key Insights
Reasoning as Inference-Time Compute
A critical finding of this research is that explicit explanations are mandatory for small model performance. When the requirement to "think" (generate an explanation) before scoring was removed, performance dropped significantly—from 21.5% to 14.4% nDCG@10. For small models, the act of generating text serves as a form of inference-time computation that facilitates complex logic.
The Role of RL
While SFT provides a strong foundation, RL allows the model to refine its logic. The researchers noted a "specialization" trade-off: while RL iterations significantly improved performance in complex domains like Math and Coding, they occasionally led to slight regressions in simpler domains. This suggests that RL helps the model prioritize deep logical connections over surface-level similarities.
Limitations and Future Work
Despite its success, InteRank faces certain constraints. Due to hardware limitations (single A100 GPU), the model was restricted to a 4K token context window, which may limit its effectiveness on very long documents. Additionally, the system's performance is inherently capped by the quality of the initial retrieval stage. Future work may involve applying similar RL alignment to the first-stage retriever to create a more robust end-to-end reasoning pipeline.
Conclusion
InteRank demonstrates that specialized training can bridge the gap between model size and reasoning capability. By incentivizing explicit reasoning through distillation and reinforcement learning, a 3B-parameter model can achieve state-of-the-art results in information retrieval, providing a scalable and efficient solution for the next generation of search and RAG systems.
Summary updated