Your Dense Retriever is Secretly an Expeditious Reasoner

Introduction

Modern search engines and Retrieval-Augmented Generation (RAG) systems increasingly rely on Dense Retrievers—models that map text into high-dimensional vector spaces. While efficient, these models often struggle with "reasoning-intensive" queries where the user's intent is implicit or complex. To bridge this gap, researchers typically employ Large Language Models (LLMs) to rewrite queries into more descriptive, reasoning-heavy versions. However, this approach introduces a significant bottleneck: LLMs are computationally expensive and introduce high latency, making them impractical for real-time production environments.

This paper introduces AdaQR (Adaptive Query Reasoning), a framework designed to capture the benefits of LLM-based reasoning without the associated overhead by training a lightweight "Dense Reasoner" to mimic LLM transformations directly in the embedding space.

The AdaQR Framework

The AdaQR framework optimizes the retrieval pipeline by introducing a hybrid approach to query reasoning. It consists of three primary components:

1. The Dense Reasoner (DR): A lightweight, two-layer Multi-Layer Perceptron (MLP) that learns to transform a standard query vector into a "reasoned" vector. It effectively predicts where an LLM-rewritten query would land in the vector space.
2. The LLM Reasoner: A high-capacity LLM (e.g., DeepSeek-R1) that serves as the teacher during training and acts as a fallback for exceptionally difficult queries.
3. The Reasoner Router (RR): A decision-making module that evaluates incoming queries to determine if they can be handled by the fast Dense Reasoner or if they require the full LLM. This routing is guided by an "oracle anchor"—a representation of queries that have historically shown predictable reasoning trajectories.

Expand

Source, p. 4

Key Insights: Reasoning as a Vector Shift

A core discovery of this work is that LLM reasoning is not a stochastic process in the embedding space. When an LLM rewrites a query, it moves the embedding in a systematic, structured direction. By visualizing these transformations using Principal Component Analysis (PCA), the authors found that queries follow consistent "trajectories." Because these shifts are structured, a simple MLP can approximate them with high fidelity. This suggests that dense retrievers possess a latent capacity for reasoning that can be unlocked through mathematical shifts rather than explicit text generation.

Experimental Results

The framework was evaluated on the BRIGHT benchmark, a challenging dataset for reasoning-intensive retrieval. AdaQR demonstrated significant efficiency gains and performance improvements across a variety of LLM configurations.

Expand

Source, p. 1

Key findings include:

• Efficiency: AdaQR reduced reasoning costs by an average of 28.12% by routing simpler queries to the Dense Reasoner.
• Accuracy: Despite the cost reduction, retrieval performance (nDCG@10) improved by an average of 7.24%. In specific configurations using ReasonIR-8B and DeepSeek-R1, performance jumped by as much as 22.11%.
• Domain Strengths: While LLMs remain superior for conversational nuances, the Dense Reasoner actually outperformed full LLMs in highly structured domains like mathematics and coding (LeetCode).

Conclusion and Future Work

AdaQR proves that it is possible to achieve LLM-level reasoning in retrieval systems at a fraction of the cost. By treating reasoning as a learnable vector transformation, the framework enables "near-zero-cost" reasoning for production RAG systems. Future research will focus on extending these vector-shift techniques to multi-modal retrieval and further refining the routing mechanisms to handle more complex linguistic pragmatics.

@misc{zhang2025your,
  author = {Yichi Zhang and Jun Bai and Zhixin Cai and Shuhan Qin and Zhuofan Chen and Jinghua Guan and Wenge Rong},
  title = {{Your Dense Retriever is Secretly an Expeditious Reasoner}},
  year = {2025},
  eprint = {2510.21727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.21727},
}