QAgent: A modular Search Agent with Interactive Query Understanding

Yi Jiang, Lei Shen, Lujie Niu, Sendong Zhao, Wenbo Su, Bo ZhengarXiv 2025 · DOI · arXiv

Introduction

Retrieval-Augmented Generation (RAG) has become a cornerstone of modern Large Language Model (LLM) applications. However, traditional "retrieve-then-read" pipelines often falter when faced with complex, multi-hop queries where a single search pass is insufficient. While iterative "agentic" solutions have been proposed, they frequently suffer from "reward hacking"—a phenomenon where models learn to bypass retrieval and answer from internal memory to maximize training scores. This paper introduces QAgent, a modular search agent designed to overcome these limitations through interactive query understanding and a novel two-stage reinforcement learning (RL) strategy.

The QAgent Framework

QAgent operates through a dynamic "Plan-Search-Information-Reflect" loop. Unlike static retrievers, QAgent treats search as an active, iterative process: it formulates a plan, executes multiple queries, evaluates the returned information, and decides whether further searching is required to satisfy the user's information need.

The QAgent architecture, illustrating the iterative Plan-Search-Information-Reflect loop and the two-stage GRPO training process.

The system utilizes Group Relative Policy Optimization (GRPO), an efficient RL algorithm popularized by DeepSeek-R1. GRPO optimizes the agent's policy by comparing multiple outputs within a group, eliminating the need for a separate critic model and streamlining the training of complex agentic behaviors.

Two-Stage Training for Robust Generalization

To ensure QAgent remains a versatile "plug-and-play" module rather than a specialized, overfitted tool, the authors employ a unique two-stage training process:

  1. Stage 1 (End-to-End): The agent is trained to maximize final answer correctness. This phase establishes foundational reasoning and retrieval capabilities.
  2. Stage 2 (Generalized RL): To combat the "Information Utilization Trap" (reward hacking), the agent is trained against a frozen generator. It receives rewards only if this external, static model can answer the question using the documents the agent retrieved. This forces the agent to optimize purely for the quality and relevance of the information it fetches.

Interactive Query Understanding

A key innovation of QAgent is its ability to handle diverse query structures. While traditional systems might use simple serial or parallel query decomposition, QAgent is designed to support more sophisticated structures like Unified Query Trees and Adaptive Understanding.

Evolution of query understanding structures from simple serial/parallel models to QAgent's unified and adaptive approaches.

This flexibility allows the agent to navigate complex information landscapes, identifying when a search path has reached a dead end and adjusting its strategy in real-time, mimicking human-like research behavior.

Experimental Results

QAgent was evaluated across five open-domain Question Answering (QA) datasets, including multi-hop benchmarks like HotpotQA and Musique. The results highlight several key strengths:

  • Superior Performance: QAgent outperformed strong RL baselines like Search-R1, achieving an average 2.66% improvement in F1 score.
  • Modularity and Scaling: When integrated as a submodule for a frozen 3B generator, it improved Exact Match (EM) scores by 5.35% over naive RAG. Notably, these gains scaled even further when the agent was paired with larger 7B models, demonstrating its effectiveness as a universal search specialist.
  • Retriever Agnostic: The agent showed consistent benefits regardless of whether the underlying search engine used traditional keyword matching (BM25) or modern vector embeddings (E5).

Conclusion

QAgent represents a significant step toward modular, intelligent search agents. By decoupling retrieval optimization from answer generation through its two-stage training, it provides a robust, high-performance solution for complex query understanding that can be seamlessly integrated into diverse AI systems without losing its effectiveness.

Summary updated

Abstract

Large language models (LLMs) excel at natural language tasks but are limited by their static parametric knowledge, especially in knowledge-intensive task. Retrieval-augmented generation (RAG) mitigates this by integrating external information. However, (1) traditional RAG struggles with complex query understanding, and (2) even search agents trained with reinforcement learning (RL), despite their promise, still face generalization and deployment challenges. To address these limitations, we propose QAgent, a unified agentic RAG framework that employs a search agent for adaptive retrieval. This agent optimizes its understanding of the query through interactive reasoning and retrieval. To facilitate real-world application, we focus on modular search agent for query understanding that are plug-and-play in complex systems. Secifically, the agent follows a multi-step decision process trained with RL to maximize retrieval quality and support accurate downstream answers. We further analyze the strengths and weaknesses of end-to-end RL and propose a strategy that focuses on effective retrieval, thereby enhancing generalization in LLM applications. Experiments show QAgent excels at QA and serves as a plug-and-play module for real-world deployment.
BibTeX
@misc{jiang2025qagent,
  author = {Yi Jiang and Lei Shen and Lujie Niu and Sendong Zhao and Wenbo Su and Bo Zheng},
  title = {{QAgent: A modular Search Agent with Interactive Query Understanding}},
  year = {2025},
  eprint = {2510.08383},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.08383},
}
Figure