Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets

Adam Younsi, Ahmed Attia, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakim Hacid, Salem LahlouSubmission to ICLR 2026 · Rejected

TL;DR

Standard reinforcement learning teaches large language models to maximize a single reward, which often causes them to collapse onto a single rigid solution path. This paper introduces an automated step-level Process Reward Model (PRM) trained via Monte Carlo Tree Search (MCTS) and uses it to guide Generative Flow Networks (GFlowNets) at the reasoning-step level. The resulting models sample diverse, high-reward mathematical solutions, simultaneously improving in-domain accuracy (+2.59% on MATH Level 5) and achieving significant gains on unseen out-of-domain benchmarks (+9.4% on SAT Math).

What Problem Does This Paper Address?

In complex mathematical reasoning, arriving at the right answer through only one memorized strategy indicates brittle pattern matching rather than genuine problem-solving. While reinforcement learning from human feedback (RLHF) methods like PPO enhance accuracy, their reward-maximization objective naturally concentrates probability mass on a single "best" path—stifling solution diversity.

Furthermore, guiding multi-step reasoning requires intermediate feedback via Process Reward Models (PRMs). Traditionally, training a PRM requires costly step-by-step human annotations. Prior automated PRMs often rely on coarse binary labels and throw away intermediate rollouts, while existing GFlowNet adaptations for language models operate at the token level, disrupting semantic coherence.

Methods

The authors propose an end-to-end framework combining automated process supervision with step-level flow networks:

  1. Automated Continuous PRM via MCTS:
  2. Uses Qwen2.5-Math to perform Monte Carlo Tree Search ($k=96$ rollouts per step).
  3. Instead of binary labels, steps are assigned continuous values $MC(s) \in [0, 1]$ representing the empirical probability that the step leads to a correct final answer (terminating search at the first identified error).
  4. Rollout Reuse & Step Similarity Grouping: Rather than discarding failed or intermediate rollouts, the authors extract individual steps and cluster them based on two criteria: mathematical calculation equivalence and Levenshtein string similarity (threshold $\ge 0.85$). This expands an initial 100k steps into a rich dataset of 2.1M steps without extra compute.
  5. Step-Level GFlowNet Fine-Tuning:
  6. Instead of operating over discrete tokens, actions correspond to generating complete reasoning steps, preserving semantic meaning and mathematical validity.
  7. Trajectory rewards are defined multiplicatively across steps: $R(s_n) = \prod_{i=1}^n U(s_i \mid s_{i-1})$, where $U$ is the PRM score.
  8. Policies are trained using the Subtrajectory Balance (SubTB($\lambda$)) objective with $\lambda=1.0$ and a prioritized replay buffer. Because GFlowNets optimize to sample trajectories with probability proportional to their reward ($P(x) \propto R(x)$), the LLM learns to explore multiple distinct, high-quality reasoning pathways.

Datasets

  • Training Data: 70,000 problems from OpenMathInstruct-2 for MCTS data generation; a 10,000-problem subset of challenging Level 5 problems was used for downstream policy fine-tuning.
  • Evaluation Benchmarks:
  • MATH Level 5 / MATH Hard (in-domain complex competition math).
  • GSM8K (grade-school math word problems).
  • SAT MATH (out-of-domain standardized test problems to test reasoning generalization).
  • Note: No new public benchmarks were introduced, but the paper created a 2.1M-step annotated PRM training dataset via automated MCTS.

Key Insights and Ideas

  • Diversity and Accuracy Are Not at Odds: PPO increases uniformity (average semantic similarity between solutions rose from 0.80 to 0.82), whereas GFlowNets decrease similarity to 0.78, proving that models can uncover alternative reasoning paths without sacrificing correctness.
  • Step-Level vs. Token-Level Control: Applying GFlowNets at the step boundary matches the natural hierarchy of human mathematical proof, avoiding the semantic fragmentation that plagues token-level flow modeling.
  • Generalization Through Multi-Path Exploration: Learning multiple diverse solution patterns led to dramatic generalization gains on out-of-distribution math tests, suggesting that learning multiple valid solution paths prevents overfitting to benchmark-specific heuristics.
  • Unexpected Efficiency: GFlowNet fine-tuning was roughly 2× faster than the PPO baseline (1 hour vs. 2 hours on 8× A100 GPUs for 10k problems) due to faster convergence and sample efficiency from the prioritized replay buffer.

Key Results

  • Accuracy Improvements:
  • Llama3.2-3B-it: MATH Level 5 rose from 14.46% (base) and 15.32% (PPO) to 17.05% (+GFlowNet). On SAT MATH, accuracy leapt from 65.6% to 75.0% (+9.4% absolute gain).
  • Llama3.1-8B-it: MATH Level 5 improved from 17.96% to 18.67%, and SAT MATH jumped from 81.2% to 84.4%.
  • PRM Quality: Outperformed open-source alternatives like Skywork-o1-Open-PRM-Qwen-2.5-7B when used to guide test-time step search.
  • Diversity Analysis: Cosine similarity across sentence embeddings (paraphrase-MiniLM-L6-v2) dropped from 0.82 (PPO) to 0.78 (GFlowNet), yielding qualitatively different approaches (e.g., distributing division before subtraction vs. subtracting first).

Limitations and Future Work

  • MCTS Upfront Overhead: Generating the PRM dataset required significant initial computation (~4 hours on 8× A100 GPUs), even though policy fine-tuning itself is fast.
  • Hyperparameter Sensitivity: The framework relies on careful tuning of the similarity-grouping threshold (0.85) and replay buffer prioritization.
  • Automated Data Biases: MCTS-based rollouts can occasionally propagate systematic reasoning blind spots into the PRM.
  • Future Work: The authors propose extending step-level GFlowNets to offline training regimes, developing domain-specific diversity metrics for formal mathematics, and applying the method to code generation and automated theorem proving.

Summary updated

Abstract

Achieving both accuracy and diverse reasoning remains challenging for Large Language Models (LLMs) in complex domains like mathematics. A key bottleneck is evaluating intermediate reasoning steps to guide generation without costly human annotations. To address this, we first introduce a novel Process Reward Model (PRM) trained automatically using Monte Carlo Tree Search coupled with a similarity-based data augmentation technique, effectively capturing step-level reasoning quality. Leveraging this PRM, we then adapt Generative Flow Networks (GFlowNets) to operate at the reasoning step level. Unlike traditional reinforcement learning focused on maximizing a single reward, GFlowNets naturally sample diverse, high-quality solutions proportional to their rewards, as measured by our PRM. Empirical evaluation shows strong improvements in both accuracy and solution diversity on challenging mathematical benchmarks (e.g., +2.59\% absolute accuracy on MATH Level 5 for Llama3.2-3B), with effective generalization to unseen datasets (+9.4\% absolute on SAT MATH). Furthermore, we benchmark our PRM against existing open-source reward models, demonstrating superior alignment with reasoning quality and more consistent guidance for downstream generation. Our work demonstrates the potential of PRM-guided, step-level GFlowNets for developing more robust and versatile mathematical reasoning in LLMs.
BibTeX
@misc{younsi2025accurate,
  author = {Adam Younsi and Ahmed Attia and Abdalgader Abubaker and Mohamed El Amine Seddik and Hakim Hacid and Salem Lahlou},
  title = {{Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets}},
  year = {2025},
  eprint = {2504.19981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.19981},
}
Figure