Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision

Xiaopeng Ye, Xu Chen, Chaoliang Zhang, Zhaocheng Du, Jun Xu, Gang Wang, Zhenhua Dong2025 · DOI

TL;DR

This paper introduces Q-PRM, a novel framework for adaptive query rewriting in Retrieval-Augmented Generation (RAG) systems. By reformulating query rewriting as a Markov Decision Process and introducing step-level process supervision, Q-PRM effectively addresses over- and under-refinement issues common in existing methods, leading to significantly improved performance across diverse query complexities.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) systems are crucial for knowledge-intensive tasks, but their effectiveness hinges on how well they handle user queries. Real-world queries vary greatly in complexity, from simple factual questions to multi-hop reasoning and ambiguous requests. To improve RAG performance, raw queries are often rewritten into more precise and retrieval-friendly forms. However, existing query rewriting methods struggle with this complexity. Outcome-supervised methods often miss critical intermediate reasoning steps for complex queries (under-refinement), while heuristic methods can add unnecessary information to simple queries (over-refinement). This paper identifies the root cause as the lack of supervision signals for intermediate rewriting steps.

Methods

Q-PRM tackles this challenge by introducing a novel framework with three key stages:

  1. Data Collection (Uncertainty-aware Monte Carlo Tree Search - MCTS): To overcome the lack of readily available process supervision signals, Q-PRM employs an uncertainty-aware MCTS approach. This method automatically collects robust supervision signals by exploring potential rewriting paths and evaluating their quality, guided by an uncertainty criterion to balance exploration and exploitation.
  2. Model Training (Reinforced Self-Training and PRM Training): The collected data is used to train two core components:
  3. Rewriter Model: An LLM is first supervised fine-tuned (SFT) on high-reward trajectories from MCTS. Then, it undergoes reinforced self-training to enhance its multi-step reasoning ability for query rewriting.
  4. Process Reward Model (PRM): The PRM is trained using a pairwise loss function to predict rewards for intermediate rewriting steps. This approach focuses on relative differences between good and bad steps, making it more robust to reward fluctuations.
  5. PRM-Guided Inference: During inference, Q-PRM combines the trained rewriter and PRM. It uses a novel decoding strategy that weights the rewriter's output probability with the PRM's predicted process reward, allowing for fine-grained guidance and adaptive query refinement.

Datasets

The paper evaluates Q-PRM on four English open-domain QA datasets known for varying query complexities:

  • HotpotQA: Multi-hop reasoning.
  • StrategyQA: Multi-hop reasoning.
  • AmbigQA: Ambiguous queries.
  • PopQA: Single-hop queries.

Additionally, an online A/B test was conducted on a commercial Chinese RAG system. For generalization experiments, they also used subsets from MRQA-NQ, DomainRAG, CORAL, and ASQA.

Key Insights and Ideas

  • The Importance of Step-level Supervision: The core insight is that supervising intermediate rewriting steps, not just the final outcome, is crucial for handling diverse query complexities. This process supervision allows for more targeted and effective guidance.
  • MCTS for Signal Generation: Using MCTS with an uncertainty-aware criterion is an effective way to automatically generate valuable process supervision signals when manual annotation is infeasible.
  • PRM for Adaptive Guidance: The Process Reward Model (PRM) acts as an intelligent guide, providing step-by-step feedback that helps the rewriter adapt its strategy based on the query's complexity.
  • Addressing Over/Under-Refinement: Q-PRM's approach directly tackles the over-refinement (adding too much detail) and under-refinement (missing crucial steps) issues plaguing previous methods.

Key Results

  • Superior Performance: Q-PRM consistently outperforms various baseline methods (heuristic and outcome-supervised) across all tested datasets and metrics (EM, F1, Acc). For instance, on the StrategyQA dataset, Q-PRM achieved a 4.15% higher EM score than baselines without PRM guidance.
  • Online A/B Test Success: In a real-world Chinese RAG system, Q-PRM achieved a 57.3% win rate against the existing production system, demonstrating its practical effectiveness.
  • Robustness: The method shows strong generalization capabilities across different RAG tasks and query types, including multi-hop, ambiguous, and single-hop queries.
  • Ablation Studies: Removing any component of Q-PRM (PRM guidance, CoT decoding, self-training, SFT) led to performance degradation, highlighting the synergistic contribution of each part.

Limitations and Future Work

  • Focus on Pre-retrieval: Q-PRM primarily focuses on query rewriting in the pre-retrieval stage. The authors suggest integrating it with iterative RAG approaches that perform multi-turn retrieval and rewriting during inference for even better handling of complex tasks.
  • Inference Overhead: Introducing a PRM for multi-step query rewriting incurs additional inference latency. Future work aims to mitigate this overhead while retaining the benefits of process supervision.
  • Alternative Reward Mechanisms: While the current PRM is effective, future research could explore rule-based reward functions within RL frameworks to further enhance stability and effectiveness, especially in light of emerging Large Reasoning Models.

Summary updated

BibTeX
@misc{ye2025q,
  author = {Xiaopeng Ye and Xu Chen and Chaoliang Zhang and Zhaocheng Du and Jun Xu and Gang Wang and Zhenhua Dong},
  title = {{Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.817},
}

Loading related content...

Figure