LimRank: Less is More for Reasoning-Intensive Information Reranking

Tingyu Song, Yilun Zhao, Siyue Zhang, Chen Zhao, Arman Cohan

Abstract

Existing approaches typically rely on large-scale fine-tuning to adapt LLMs for information reranking tasks, which is computationally expensive. In this work, we demonstrate that modern LLMs can be effectively adapted using only minimal, high-quality supervision. To enable this, we design LIMRANK-SYNTHESIZER, a reusable and open-source pipeline for generating diverse, challenging, and realistic reranking examples. Using this synthetic data, we fine-tune our reranker model, LIMRANK. We evaluate LIMRANK on two challenging benchmarks, i.e., BRIGHT for reasoning-intensive retrieval and FollowIR for instruction-following retrieval. Our experiments demonstrate that LIMRANK achieves competitive performance, while being trained on less than 5% of the data typically used in prior work. Further ablation studies demonstrate the effectiveness of LIMRANK-SYNTHESIZER and the strong generalization capabilities of LIMRANK across downstream tasks, including scientific literature search and retrieval-augmented generation for knowledge-intensive problem solving.

Introduction

Information retrieval has evolved beyond simple keyword matching. Modern "reasoning-intensive" tasks require models to understand multi-step inferences, implicit relationships, and complex instructions. Traditionally, achieving high performance in these areas has required massive supervised fine-tuning (SFT). For instance, previous state-of-the-art models like Rank1 were trained on nearly 700,000 examples, a process that is both computationally expensive and dependent on vast quantities of high-quality reasoning traces.

LIMRANK challenges this "more is better" philosophy. By applying the principle that high-quality, diverse data is a much stronger signal for Large Language Models (LLMs) than sheer volume, the authors demonstrate that a 7B model fine-tuned on just 20,000 high-quality examples can outperform models trained on 35 times more data.

Methodology: The LIMRANK-SYNTHESIZER

The core innovation is LIMRANK-SYNTHESIZER, a "bottom-up" pipeline designed to generate diverse and challenging training data. Unlike traditional methods that rely on existing document corpora, this pipeline synthesizes scenarios where relevance is often indirect and requires logical deduction.

The LIMRANK-SYNTHESIZER pipeline and the concept of indirect relevance in reasoning-intensive reranking.

The process consists of three primary stages:

  1. Query Expansion via Personas: Simple queries are rewritten into complex, nuanced questions by adopting specific "personas" (e.g., a scientist or a specialist). This creates realistic, "messy" queries that mirror real-world expert searches.
  2. Positive and Negative Passage Generation: The pipeline uses Chain-of-Thought (CoT) prompting to describe ideal "indirectly relevant" documents and then generates them. It also produces "hard negatives"—passages that appear relevant on the surface but contain subtle logical flaws or errors.
  3. LLM-Based Filtering: A high-capacity model (like DeepSeek-R1) judges the relevance of generated pairs, ensuring only high-quality, consistent examples remain in the final 20,000-sample dataset.

Experimental Results

LIMRANK was evaluated against several benchmarks, most notably BRIGHT (reasoning-intensive retrieval) and FOLLOWIR (instruction-following).

Performance comparison of LIMRANK against baseline and state-of-the-art models on the BRIGHT benchmark across various domains.

Key performance highlights include:

  • BRIGHT Benchmark: LIMRANK achieved an average 28.0 nDCG@10, surpassing Rank1-7B (27.5) and significantly outperforming the base Qwen2.5-7B (22.6). It showed particular strength in biology, economics, and sustainability domains.
  • Instruction Following: On the FOLLOWIR benchmark, it achieved a 1.2 p-MRR, a substantial lead over Rank1-7B (0.3), demonstrating a superior ability to adhere to complex search constraints.
  • Generalization: The model also excelled in scientific literature search (LitSearch) and Retrieval-Augmented Generation (RAG) tasks, reaching 30.3% accuracy on the GPQA science benchmark.

Conclusion

LIMRANK proves that for reasoning-intensive reranking, data quality and diversity are more critical than quantity. By training on less than 3% of the data used by its primary competitors, LIMRANK achieves state-of-the-art results while remaining computationally efficient. This "less is more" approach suggests that the latent reasoning capabilities of frontier LLMs can be unlocked with highly strategic, persona-driven synthetic data. Future work will focus on reducing the cost of data curation and exploring these principles in non-pointwise reranking architectures.

Summary updated

Abstract

Existing approaches typically rely on large-scale fine-tuning to adapt LLMs for information reranking tasks, which is computationally expensive. In this work, we demonstrate that modern LLMs can be effectively adapted using only minimal, high-quality supervision. To enable this, we design LIMRANK-SYNTHESIZER, a reusable and open-source pipeline for generating diverse, challenging, and realistic reranking examples. Using this synthetic data, we fine-tune our reranker model, LIMRANK. We evaluate LIMRANK on two challenging benchmarks, i.e., BRIGHT for reasoning-intensive retrieval and FollowIR for instruction-following retrieval. Our experiments demonstrate that LIMRANK achieves competitive performance, while being trained on less than 5% of the data typically used in prior work. Further ablation studies demonstrate the effectiveness of LIMRANK-SYNTHESIZER and the strong generalization capabilities of LIMRANK across downstream tasks, including scientific literature search and retrieval-augmented generation for knowledge-intensive problem solving.
BibTeX
@inproceedings{song2025limrank,
  author = {Tingyu Song and Yilun Zhao and Siyue Zhang and Chen Zhao and Arman Cohan},
  title = {{LimRank: Less is More for Reasoning-Intensive Information Reranking}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure