ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

Jeff Duan, Matt ThomsonarXiv 2026 · arXiv

TL;DR

ROAM-GRPO is a novel training augmentation designed to stop Large Language Models (LLMs) from getting "stuck" on a single way of solving a problem. By detecting well-solved tasks and routing them to an "off-mode" track that forces the model to find alternative valid strategies, the researchers increased reasoning diversity and boosted performance on difficult math benchmarks by up to 4.7% in multi-attempt settings (pass@16).

What Problem Does This Paper Address?

When we train LLMs on complex reasoning tasks using Reinforcement Learning with Verifiable Rewards (RLVR)—like the popular GRPO algorithm used in models like DeepSeek-R1—we often encounter mode collapse.

The model discovers one reliable way to get the right answer and pours all its "probability mass" into that single path. While this might improve the model's first-try accuracy (pass@1), it kills creativity. If the model's first attempt fails on a truly difficult problem, its next ten attempts will likely fail in the exact same way because they are nearly identical. This lack of diversity prevents models from effectively "searching" for solutions in high-stakes domains like competition math or formal code verification.

Methods

The authors introduce ROAM-GRPO (Routed Off-mode Approach Mining), which adds a "diversity engine" to the standard GRPO pipeline.

Figure 1: ROAM-GRPO training pipeline. Well-solved prompts (≥k of G completions correct) are routed off the GRPO objective onto an off-mode track that self-distills valid, distinct alternative solutions measured by τ, ϕ (abstract functions, instantiated as an LLM judge here in practice).

The process works in four main steps:

  1. Routing: During training, if a model solves a problem consistently (e.g., all 8 samples in a group are correct), the router flags it as "well-solved" and moves it to the ROAM track.
  2. Off-mode Mining: Instead of just practicing what it already knows, the model is shown its own common solution ("Method 1") and specifically prompted to generate a "genuinely different approach" ("Method 2").
  3. Verification: An external LLM judge (like Qwen3.5-9b) evaluates these new attempts for two things: Logical Validity ($\tau$) and Diversity ($\phi$). It ensures the new method isn't just a paraphrase but a different mathematical strategy.
  4. Self-Distillation: Valid and diverse new solutions are distilled back into the model using a standard cross-entropy loss, alongside an "anchor" solution to ensure the model doesn't forget the original method.

Datasets

  • Training: A 1,000-problem subset of the MATH dataset, specifically restricted to "Level 5" (the hardest) problems to ensure the tasks were complex enough to support multiple solution paths.
  • Evaluation:
    • MATH (Level 5): 400 test problems.
  • Omni-MATH: A more challenging dataset of 400 Olympiad-level problems (Levels 5-6) designed to test the limits of mathematical reasoning.

Key Insights and Ideas

  • The Pass@1 vs. Pass@k Trade-off: ROAM-GRPO often results in a lower pass@1 accuracy compared to baseline GRPO. However, it maintains or improves pass@16. This suggests that while the model might be slightly less "certain" on its first try, its overall "knowledge surface area" is much larger.
  • Delayed Activation is Key: The most effective strategy was "Delayed ROAM," where the diversity training only starts after the model has already reached a high baseline of performance via standard GRPO. This allows the model to learn how to solve problems before being forced to find alternative ways to solve them.
  • Semantic vs. Surface Diversity: The researchers used "solution dispersion" (measuring the distance between vector embeddings of different answers) to show that ROAM-GRPO models produce outputs that are fundamentally different, not just reworded.

Key Results

The researchers tested their approach using a Qwen3.5-2b model. Key findings include:

  • Performance Boost: On the difficult Omni-MATH set, Delayed ROAM-GRPO achieved a 4.7% increase in peak pass@16 over the GRPO baseline.
  • Increased Dispersion: The "solution dispersion" (a proxy for diversity) was consistently 1.5x to 2x higher than the baseline, as shown in the training dynamics.
Figure 2: pass@1, pass@16, and solution dispersion over training (MATH-L5, left; Omni-MATH L5/6, right). GRPO, ROAM-GRPO, and delayed ROAM-GRPO (activated at step 5000, dotted). Error bars on pass@1, pass@16 display 95% percentile bootstrap confidence intervals. ROAM trades pass@1 for approximately equal pass@16 (and higher dispersion). Delayed ROAM reaches the highest peak pass@16 on Omni-MATH. pass@4, pass@8 included in App. B.
  • Method-Level Variety: Using Claude Opus 5.5 as a judge, the authors confirmed that ROAM-GRPO produced a higher average number of distinct mathematical methods per problem.
Figure 4: Method-level diversity at step 7000 using an LLM judge (Claude Opus 5.5). Top: average number of distinct solution methods per problem, among correct completions only (a) and all completions (b). Bottom: fraction of pairs (within the same method bin) judged to be meaningful variants rather than surface level rewrites, among correct completions (c) and all completions (d). Error bars show 95% bootstrap confidence intervals over problems, and ∆shows ROAM −GRPO difference along with the 95% CI. All judged problems require that both arms produce at least one correct completion.

Limitations and Future Work

  • Training Stability: The authors noted that ROAM-GRPO training can be unstable, with performance fluctuating in later steps.
  • Computational Cost: Relying on an LLM judge for validity and diversity adds significant overhead to the training process. Future work could explore using deterministic verifiers (like Lean or Python interpreters) to replace the LLM judge.
  • Scalability: The study focused on a 2B-parameter model. It remains to be seen how these diversity incentives scale to much larger models that may already have more inherent "memorized" diversity.

Summary updated

Abstract

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.
BibTeX
@misc{duan2026roam,
  author = {Jeff Duan and Matt Thomson},
  title = {{ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining}},
  year = {2026},
  eprint = {2611.25926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25926},
}
Figure