Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation

Yuelyu Ji, Rui Meng, Zhuochun Li, Daqing HearXiv 2025 · arXiv · OpenReview

Abstract

Retrieval-augmented generation (RAG) grounds large language models (LLMs) in up-to-date external evidence, yet existing multi-hop RAG pipelines still issue redundant subqueries, explore too shallowly, or wander through overly long search chains. We introduce EVO-RAG, a curriculum-guided reinforcement learning framework that evolves a query-rewriting agent from broad early-stage exploration to concise late-stage refinement. EVO-RAG couples a seven-factor, step-level reward vector (covering relevance, redundancy, efficiency, and answer correctness) with a time-varying scheduler that reweights these signals as the episode unfolds. The agent is trained with Direct Preference Optimization over a multi-head reward model, enabling it to learn when to search, backtrack, answer, or refuse. Across four multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle), EVO-RAG boosts Exact Match by up to 4.6 points over strong RAG baselines while trimming average retrieval depth by 15%.

TL;DR

EVO-RAG is a curriculum-guided reinforcement learning framework designed to make multi-hop Retrieval-Augmented Generation (RAG) both more accurate and more efficient. By using a two-stage training process and a dynamic "gearbox" of seven distinct reward signals, it teaches AI agents to transition from broad information discovery to concise, evidence-backed answering, boosting accuracy by up to 4.6 points while reducing retrieval steps by 15%.

The Challenge of Multi-Hop Retrieval

Retrieval-Augmented Generation (RAG) is the standard for grounding LLMs in external facts, but "multi-hop" questions—those requiring multiple steps of reasoning (e.g., "Who is the CEO of the company that built the rocket launched yesterday?")—remain a major hurdle.

Current multi-hop RAG pipelines suffer from three main "pathologies":

  1. Redundancy: Issuing the same or similar sub-queries repeatedly.
  2. Shallow Exploration: Giving up before finding all necessary "hops" of information.
  3. Wandering: Getting lost in over-long search chains that increase latency and cost without improving the answer.

Existing Reinforcement Learning (RL) approaches for RAG often use static, "phase-agnostic" rewards. They treat the first search step the same as the last, failing to guide the agent through the natural transition from finding information to refining an answer.

Methods: The EVO-RAG Framework

The authors introduce EVO-RAG, which treats query rewriting as a sequential decision-making task. The core innovations include:

Seven-Factor Reward Vector

Instead of a single "correct/incorrect" score, the model receives a dense reward vector covering:

  • Retrieval Bonus: Did we find a "gold" document?
  • Sub-query Overlap Penalty: Is this query too similar to a previous one?
  • Step Cost & Retrieval Action Penalty: Discouraging unnecessarily long chains.
  • Backtrack Penalty: Penalizing inefficient "undo" moves.
  • Refusal Reward: Rewarding the model for truthfully stating it can't answer based on current evidence (to prevent hallucinations).
  • Answer Correctness: The final EM/F1 score.

Two-Stage Curriculum

Training is split into two distinct phases to mirror the natural progression of a search task:

The Discovery stage focuses on broad exploration to find relevant documents.
  1. Discovery Stage: Prioritizes broad exploration and finding documents.
The Refinement stage prioritizes efficiency and answer accuracy.
  1. Refinement Stage: Prioritizes efficiency, accuracy, and conciseness.

Time-Based Scheduler

Within a single episode, the weights of the rewards shift dynamically. For example, the "Step Penalty" increases as the episode progresses, acting as a "gearbox" that pressures the agent to wrap up its search.

Multi-Head Preference Model

The agent is trained using Direct Preference Optimization (DPO) over a multi-head reward model, which allows it to learn from nuanced trade-offs between different reward components.

Datasets and Evaluation

The researchers evaluated EVO-RAG across four prominent multi-hop QA benchmarks:

  • HotpotQA: Factoid-based multi-hop questions.
  • 2WikiMultiHopQA: Structured multi-hop reasoning.
  • MuSiQue: Complex, compositional multi-hop questions.
  • Bamboogle: Adversarial multi-hop questions designed to be difficult for search engines.

Notably, the models were trained only on HotpotQA to test their ability to generalize to different styles of questions in the other three datasets.

Key Insights

  • Phase-Awareness is Key: A search agent's priorities should change over time. Early on, it should be "brave" and explore; later, it should be "cautious" and focus on the final answer. EVO-RAG’s dynamic scheduler captures this intuition mathematically.
  • Multi-Head Rewards Stabilize Training: Breaking the reward into seven specific signals provides "dense" feedback. The authors found that a multi-head reward model achieved lower validation loss and more stable policy updates than a single-head model.
  • The "Refusal" Signal Curbs Hallucination: By explicitly rewarding the model for refusing to answer when evidence is insufficient, EVO-RAG learns to avoid the common LLM trap of "guessing" when the search fails.
  • Efficiency Doesn't Have to Hurt Accuracy: By penalizing sub-query overlap, the model learns to ask more diverse, informative questions, which actually leads to better answers in fewer steps.

Results

  • Accuracy Boost: EVO-RAG outperformed strong baselines (like RAG-Gym and IRCoT) across all datasets, improving Exact Match (EM) scores by up to 4.6 points.
  • Efficiency Gains: The model trimmed the average retrieval depth by 15%, proving it could find better answers with fewer API calls.
  • Generalization: Despite being trained only on HotpotQA, the agent showed significant gains on MuSiQue and Bamboogle, suggesting the learned "search strategy" is robust across domains.
  • Qualitative Success: Case studies showed that while baseline models often hallucinated or got stuck in loops, EVO-RAG used its "Backtrack" and "Refusal" actions to navigate tricky, unanswerable, or noisy scenarios.

Limitations and Future Work

  • Manual Tuning: The reward weights and the scheduling "gearbox" were manually tuned. Future work could use meta-learning to automate this tuning.
  • Automatic Metrics: The study relies on EM and F1 scores, which don't capture human nuances like "helpfulness" or "readability."
  • Scale: Experiments were limited to 8B parameter models. Scaling to larger models remains an open challenge.
  • Latent Actions: Moving toward latent action policies could further improve flexibility and reasoning depth.

Summary updated

Abstract

Retrieval-augmented generation (RAG) grounds large language models (LLMs) in up-to-date external evidence, yet existing multi-hop RAG pipelines still issue redundant subqueries, explore too shallowly, or wander through overly long search chains. We introduce EVO-RAG, a curriculum-guided reinforcement learning framework that evolves a query-rewriting agent from broad early-stage exploration to concise late-stage refinement. EVO-RAG couples a seven-factor, step-level reward vector (covering relevance, redundancy, efficiency, and answer correctness) with a time-varying scheduler that reweights these signals as the episode unfolds. The agent is trained with Direct Preference Optimization over a multi-head reward model, enabling it to learn when to search, backtrack, answer, or refuse. Across four multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle), EVO-RAG boosts Exact Match by up to 4.6 points over strong RAG baselines while trimming average retrieval depth by 15 %. Ablation studies confirm the complementary roles of curriculum staging and dynamic reward scheduling. EVO-RAG thus offers a general recipe for building reliable, cost-effective multi-hop RAG systems.
BibTeX
@misc{ji2025curriculum,
  author = {Yuelyu Ji and Rui Meng and Zhuochun Li and Daqing He},
  title = {{Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation}},
  year = {2025},
  eprint = {2505.17391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17391},
}

Loading related content...

Figure