Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking

Chris Samarinas, Hamed Zamani

TL;DR

InteRank is a novel framework that enables a compact 3B-parameter language model to outperform models 20 times its size in complex, reasoning-heavy document re-ranking. By combining synthetic data distillation from a 70B teacher with reinforcement learning (RL) to reward high-quality explanations, the authors achieve state-of-the-art results on the challenging BRIGHT benchmark.

What Problem Does This Paper Address?

Modern search engines and RAG (Retrieval-Augmented Generation) systems are increasingly asked to handle "reasoning-intensive" queries—questions that require deep domain knowledge, logical deduction, or multi-step analysis (e.g., "Why does this specific coding error occur?").

Traditional ranking models (like BM25 or standard dense retrievers) excel at keyword or semantic matching but often fail when the relationship between a query and a document is nuanced. While Large Language Models (LLMs) like GPT-4 or Llama-70B have the "brainpower" to solve these, they are too computationally expensive and slow for the high-throughput requirements of real-time document re-ranking. The challenge is: How do we distill "big model" reasoning into a "small model" footprint without losing accuracy?

Methods

The authors introduce InteRank, a three-phase pipeline to train a 3B-parameter student model (Llama 3.2 3B):

  1. Synthetic Data Generation: Instead of relying on expensive human labels, the authors scraped 20,000 question-answer pairs from 186 StackExchange communities. They used a "teacher" LLM (Llama 3.3 70B) to generate step-by-step explanations and relevance labels (0, 1, or 2) for these pairs.
  2. Knowledge Distillation (SFT): The 3B student model is first trained via Supervised Fine-Tuning (SFT) to mimic the teacher’s reasoning paths and final relevance scores.
  3. Reinforcement Learning (RL) Refinement: To move beyond simple imitation, the authors used RL (specifically a variant of the REINFORCE algorithm) to reward the model for generating better explanations. A separate reward model (Skywork-Reward) evaluates the quality of the student's reasoning. This allows the model to explore multiple valid ways to justify a ranking decision.

The Architecture: InteRank acts as a second-stage re-ranker. It takes the top 100 documents from an initial retriever (like BM25 or GTE), generates a "Chain-of-Thought" explanation for each, and outputs a discrete relevance score.

Datasets

  • Training: A custom dataset of 20K examples derived from StackExchange (coding, math, science).
  • Evaluation: The BRIGHT benchmark, a recent and difficult dataset specifically designed for reasoning-intensive retrieval. It spans domains like Biology, Earth Science, Economics, and Theorem Proving.

Key Insights and Ideas

  • Reasoning as Computation: The most critical finding is that explanations are mandatory. When the authors removed the requirement for the model to "think" (explain) before scoring, performance plummeted (e.g., from 21.5% to 14.4% nDCG@10). The act of generating text acts as "inference-time compute" that helps the small model process the logic.
  • RL Prevents "Stochastic Parrots": Distillation alone (SFT) provides a good baseline, but RL allows the model to refine its logic. Interestingly, the authors found that while more RL iterations helped in complex domains (Math, Coding), it could lead to slight regressions in simpler domains—a "specialization" trade-off.
  • Efficiency Wins: The 3B InteRank model ranked 3rd on the BRIGHT leaderboard, beating 70B-parameter models and complex ensembles. This proves that for specific tasks like ranking, a specialized small model can be more effective than a general-purpose giant.

Key Results

  • State-of-the-Art Performance: InteRank (3B) achieved an average nDCG@10 of 27.4% on BRIGHT when paired with a Qwen1.5-7B retriever.
  • Size Defiance: It outperformed Reason-to-Rank (8B) and matched or exceeded the performance of its own 70B teacher in zero-shot settings.
  • Statistical Significance: The model showed significant gains over traditional cross-encoders (like MiniLM) and dense retrievers (like BGE or E5), which often scored below 20% on the same tasks.

Limitations and Future Work

  • Context Constraints: Due to hardware limits (single A100 GPU), the model was restricted to a 4K token context window. This might limit its ability to reason over very long documents.
  • First-Stage Dependency: The model's final performance is still capped by the quality of the initial retrieval (the "top 100"). The authors suggest that applying similar RL alignment to the first-stage retriever could lead to even better end-to-end systems.
  • Reward Model Bias: The system relies on a pre-trained reward model; any biases or inaccuracies in that reward model will inevitably be baked into the student.

Summary updated

Abstract

We present a novel approach for training small language models for reasoning-intensive document ranking that combines knowledge distillation with reinforcement learning optimization. While existing methods often rely on expensive human annotations or large black-box language models, our methodology leverages web data and a teacher LLM to automatically generate high-quality training examples with relevance explanations. By framing document ranking as a reinforcement learning problem and incentivizing explicit reasoning capabilities, we train a compact 3B parameter language model that achieves state-of-the-art performance on the BRIGHT benchmark. Our model ranks third on the leaderboard while using substantially fewer parameters than other approaches, outperforming models that are over 20 times larger. Through extensive experiments, we demonstrate that generating explanations during inference, rather than directly predicting relevance scores, enables more effective reasoning with smaller language models. The self-supervised nature of our method offers a scalable and interpretable solution for modern information retrieval systems.
BibTeX
@inproceedings{samarinas2025distillation,
  author = {Chris Samarinas and Hamed Zamani},
  title = {{Distillation and Refinement of Reasoning in Small Language Models for Document Re-ranking}},
  booktitle = {ICTIR},
  year = {2025},
  doi = {10.1145/3731120.3744613},
}

Loading related content...

Figure