Introduction
Retrieval-Augmented Generation (RAG) has become a cornerstone of modern Large Language Model (LLM) applications. However, traditional "retrieve-then-read" pipelines often falter when faced with complex, multi-hop queries where a single search pass is insufficient. While iterative "agentic" solutions have been proposed, they frequently suffer from "reward hacking"—a phenomenon where models learn to bypass retrieval and answer from internal memory to maximize training scores. This paper introduces QAgent, a modular search agent designed to overcome these limitations through interactive query understanding and a novel two-stage reinforcement learning (RL) strategy.
The QAgent Framework
QAgent operates through a dynamic "Plan-Search-Information-Reflect" loop. Unlike static retrievers, QAgent treats search as an active, iterative process: it formulates a plan, executes multiple queries, evaluates the returned information, and decides whether further searching is required to satisfy the user's information need.
The system utilizes Group Relative Policy Optimization (GRPO), an efficient RL algorithm popularized by DeepSeek-R1. GRPO optimizes the agent's policy by comparing multiple outputs within a group, eliminating the need for a separate critic model and streamlining the training of complex agentic behaviors.
Two-Stage Training for Robust Generalization
To ensure QAgent remains a versatile "plug-and-play" module rather than a specialized, overfitted tool, the authors employ a unique two-stage training process:
- Stage 1 (End-to-End): The agent is trained to maximize final answer correctness. This phase establishes foundational reasoning and retrieval capabilities.
- Stage 2 (Generalized RL): To combat the "Information Utilization Trap" (reward hacking), the agent is trained against a frozen generator. It receives rewards only if this external, static model can answer the question using the documents the agent retrieved. This forces the agent to optimize purely for the quality and relevance of the information it fetches.
Interactive Query Understanding
A key innovation of QAgent is its ability to handle diverse query structures. While traditional systems might use simple serial or parallel query decomposition, QAgent is designed to support more sophisticated structures like Unified Query Trees and Adaptive Understanding.
This flexibility allows the agent to navigate complex information landscapes, identifying when a search path has reached a dead end and adjusting its strategy in real-time, mimicking human-like research behavior.
Experimental Results
QAgent was evaluated across five open-domain Question Answering (QA) datasets, including multi-hop benchmarks like HotpotQA and Musique. The results highlight several key strengths:
- Superior Performance: QAgent outperformed strong RL baselines like Search-R1, achieving an average 2.66% improvement in F1 score.
- Modularity and Scaling: When integrated as a submodule for a frozen 3B generator, it improved Exact Match (EM) scores by 5.35% over naive RAG. Notably, these gains scaled even further when the agent was paired with larger 7B models, demonstrating its effectiveness as a universal search specialist.
- Retriever Agnostic: The agent showed consistent benefits regardless of whether the underlying search engine used traditional keyword matching (BM25) or modern vector embeddings (E5).
Conclusion
QAgent represents a significant step toward modular, intelligent search agents. By decoupling retrieval optimization from answer generation through its two-stage training, it provides a robust, high-performance solution for complex query understanding that can be seamlessly integrated into diverse AI systems without losing its effectiveness.
Summary updated