Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width

Zheng Liu, Chaofan Li, Shitao Xiao, Chaozhuo Li, Defu Lian, Yingxia ShaoarXiv 2025 · DOI · arXiv

Introduction

In modern information retrieval and Retrieval-Augmented Generation (RAG) pipelines, re-ranking serves as a critical step where Large Language Models (LLMs) evaluate a subset of documents to ensure high precision. While LLMs excel at this task, their deployment is often hindered by significant computational costs and high latency. Traditional acceleration methods, such as ad-hoc pruning, require static model versions tailored to specific hardware or latency constraints. This lack of flexibility necessitates a complete retraining or fine-tuning cycle whenever requirements change. The Matryoshka Re-Ranker addresses this by introducing a flexible architecture that allows for runtime adjustments to model depth and width without retraining.

Methods

The Matryoshka Re-Ranker architecture embeds multiple smaller "sub-models" within a single large LLM backbone (e.g., Mistral-7B). This is achieved through three primary innovations:

  1. Flexible Architecture (Depth and Width): The model is trained to produce ranking scores from any intermediate layer (depth) and supports sequence length compression (width) via importance-aware merging of attention weights.
  2. Cascaded Self-Distillation: To maintain the accuracy of smaller sub-structures, the full-scale model acts as a teacher. During a single efficient training pass, smaller configurations are taught to mimic the outputs of their larger super-networks.
  3. Factorized Compensation (V-LoRA & H-LoRA): To recover precision lost during compression, the authors utilize a novel Low-Rank Adaptation (LoRA) strategy. By splitting compensation into Vertical LoRA (for depth) and Horizontal LoRA (for width), the model can support thousands of configurations through simple linear addition of these factors at runtime.
Comparison between the Matryoshka Re-ranker's flexible configurations and traditional ad-hoc pruning methods.

Datasets

The model's performance was validated across several industry-standard benchmarks:

  • MSMARCO (Passage and Document): Standard web search ranking tasks.
  • BEIR Benchmark: A diverse collection of 14 datasets spanning domains such as finance, bio-medicine, and social media.
  • TREC Deep Learning (2019 & 2020): Focused on high-precision evaluation at the top of the results list.

Key Insights

  • The Nesting Property: The architecture ensures that sub-structures are dominated by their super-architectures, meaning performance scales predictably as more layers or sequence width are added.
  • Runtime Adaptability: Developers can dynamically adjust the model's "volume knob" to prioritize either speed or accuracy based on real-time server load or hardware availability.
  • Efficiency without Sacrifice: A "lightweight" configuration (16 layers and 50% width) achieved results within 0.1% of the full 32-layer Mistral-7B model while delivering a 2x speedup.

Key Results

The Matryoshka Re-Ranker demonstrates state-of-the-art performance, outperforming established models like RankLlama and RankGPT-4 on various BEIR tasks. Empirical results show that the model can save over 60% of FLOPs and achieve significant inference time reductions with negligible loss in ranking accuracy (MRR@10). Even when reduced to only 6 layers, the model remains a superior re-ranker compared to traditional first-stage retrievers like BM25.

Performance curves showing the relationship between MRR@10 and computational savings (FLOPs and inference time).

Limitations and Future Work

While providing unprecedented flexibility, the initial training process is more complex than standard fine-tuning due to the multi-objective distillation required across sub-structures. Additionally, performance remains inherently tied to the underlying LLM backbone. Future research may explore using this architecture as a "super teacher" to distill even more efficient, non-LLM scale models.

Summary updated

Abstract

Large language models (LLMs) provide powerful foundations to perform fine-grained text re-ranking. However, they are often prohibitive in reality due to constraints on computation bandwidth. In this work, we propose a \textbf{flexible} architecture called \textbf{Matroyshka Re-Ranker}, which is designed to facilitate \textbf{runtime customization} of model layers and sequence lengths at each layer based on users' configurations. Consequently, the LLM-based re-rankers can be made applicable across various real-world situations. The increased flexibility may come at the cost of precision loss. To address this problem, we introduce a suite of techniques to optimize the performance. First, we propose \textbf{cascaded self-distillation}, where each sub-architecture learns to preserve a precise re-ranking performance from its super components, whose predictions can be exploited as smooth and informative teacher signals. Second, we design a \textbf{factorized compensation mechanism}, where two collaborative Low-Rank Adaptation modules, vertical and horizontal, are jointly employed to compensate for the precision loss resulted from arbitrary combinations of layer and sequence compression. We perform comprehensive experiments based on the passage and document retrieval datasets from MSMARCO, along with all public datasets from BEIR benchmark. In our experiments, Matryoshka Re-Ranker substantially outperforms the existing methods, while effectively preserving its superior performance across various forms of compression and different application scenarios.
BibTeX
@misc{liu2025matryoshka,
  author = {Zheng Liu and Chaofan Li and Shitao Xiao and Chaozhuo Li and Defu Lian and Yingxia Shao},
  title = {{Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width}},
  year = {2025},
  eprint = {2501.16302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.16302},
}
Figure