Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets

Adam Younsi, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakim Hacid, Salem Lahlou

Abstract

Achieving both accuracy and diverse reasoning remains challenging for Large Language Models (LLMs) in complex domains like mathematics. A key bottleneck is evaluating intermediate reasoning steps to guide generation without costly human annotations. To address this, we first introduce a novel Process Reward Model (PRM) trained automatically using Monte Carlo Tree Search coupled with a similarity-based data augmentation technique, effectively capturing step-level reasoning quality. Leveraging this PRM, we then adapt Generative Flow Networks (GFlowNets) to operate at the reasoning step level. Unlike traditional reinforcement learning focused on maximizing a single reward, GFlowNets naturally sample diverse, high-quality solutions proportional to their rewards, as measured by our PRM. Empirical evaluation shows strong improvements in both accuracy and solution diversity on challenging mathematical benchmarks (e.g., +2.59% absolute accuracy on MATH Level 5 for Llama3.2-3B), with effective generalization to unseen datasets (+9.4% absolute on SAT MATH). Furthermore, we benchmark our PRM against existing open-source reward models, demonstrating superior alignment with reasoning quality and more consistent guidance for downstream generation. Our work demonstrates the potential of PRM-guided, step-level GFlowNets for developing more robust and versatile mathematical reasoning in LLMs.

Introduction

Current LLMs often struggle with complex, multi-step reasoning. While they have become better at reaching the correct final answer, they tend to be "one-trick ponies." Traditional Reinforcement Learning (RL) methods like PPO (Proximal Policy Optimization) often suffer from mode collapse, where the model learns one specific way to solve a problem and exploits it to maximize reward, ignoring other valid strategies.

Furthermore, training models to evaluate their own intermediate steps—a "Process Reward Model" (PRM)—usually requires massive amounts of expensive human-annotated data. This research addresses two critical questions:

  1. How can we train a high-quality PRM without human labels?
  2. How can we use that PRM to encourage the model to explore diverse reasoning paths rather than just memorizing one?

Methodology

The framework consists of a two-stage process: automated PRM training followed by step-level GFlowNet fine-tuning.

Automated PRM Training

To eliminate the need for human labeling, the authors employ Monte Carlo Tree Search (MCTS). For any given reasoning step, 96 "rollouts" (completing the problem from that step) are performed to determine how often it leads to a correct answer. This provides a continuous quality score (e.g., a 0.85 score if 85% of rollouts succeed).

To scale this efficiently, a similarity-based grouping technique is introduced. If two reasoning steps are semantically similar (calculated via Levenshtein distance and calculation consistency), they share their rollout data. This augmentation turned 100k raw MCTS examples into a robust dataset of 2.1 million entries.

Similarity-based data augmentation for PRM training

Step-Level GFlowNets

Unlike standard RL that maximizes the expected reward, GFlowNets are designed to sample solutions with a probability proportional to their reward. This paper operates at the level of complete reasoning steps rather than individual tokens to avoid "semantic fragmentation." They utilize the Subtrajectory Balance (SubTB) loss function, which allows the model to learn from incomplete or partial reasoning paths, enhancing training stability and efficiency.

Experimental Results

The models were trained on 10,000 questions from the OpenMathInstruct2 dataset and evaluated on benchmarks including MATH Hard, GSM8K, and SAT MATH.

Accuracy and Generalization

The Llama 3.2-3B model saw an absolute accuracy increase of 2.59% on MATH Level 5 and 4.4% on SAT MATH. Notably, the 3B model's performance on SAT MATH jumped from 65.6% (baseline) to 75.0% (GFlowNet), significantly outperforming the PPO-tuned version (70.0%).

Performance comparison of GFlowNet vs PPO and baselines

Diversity and Robustness

A key insight is that training for diversity via GFlowNets leads to better generalization. By learning multiple ways to solve a problem, the model captures fundamental mathematical patterns that transfer better to new datasets compared to models that only learned one "optimal" path. Using semantic embedding similarity, the authors demonstrated that GFlowNet solutions are significantly more distinct from one another than those produced by PPO.

Conclusion and Future Work

This work demonstrates that step-level GFlowNets, guided by an automatically trained PRM, can significantly enhance both the accuracy and diversity of LLM mathematical reasoning. While the initial MCTS data generation is computationally expensive, it is a one-time offline cost that yields a highly effective reward signal. Future research may extend this approach to other complex domains such as program synthesis and formal theorem proving, where intermediate steps are well-defined and solution diversity is highly valued.

Summary updated

Abstract

Achieving both accuracy and diverse reasoning remains challenging for Large Language Models (LLMs) in complex domains like mathematics. A key bottleneck is evaluating intermediate reasoning steps to guide generation without costly human annotations. To address this, we first introduce a novel Process Reward Model (PRM) trained automatically using Monte Carlo Tree Search coupled with a similarity-based data augmentation technique, effectively capturing step-level reasoning quality. Leveraging this PRM, we then adapt Generative Flow Networks (GFlowNets) to operate at the reasoning step level. Unlike traditional reinforcement learning focused on maximizing a single reward, GFlowNets naturally sample diverse, high-quality solutions proportional to their rewards, as measured by our PRM. Empirical evaluation shows strong improvements in both accuracy and solution diversity on challenging mathematical benchmarks (e.g., +2.59% absolute accuracy on MATH Level 5 for Llama3.2-3B), with effective generalization to unseen datasets (+9.4\% absolute on SAT MATH). Furthermore, we benchmark our PRM against existing open-source reward models, demonstrating superior alignment with reasoning quality and more consistent guidance for downstream generation. Our work demonstrates the potential of PRM-guided, step-level GFlowNets for developing more robust and versatile mathematical reasoning in LLMs.
BibTeX
@misc{younsi2025accurate,
  author = {Adam Younsi and Abdalgader Abubaker and Mohamed El Amine Seddik and Hakim Hacid and Salem Lahlou},
  title = {{Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNets}},
  year = {2025},
  eprint = {2504.19981},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.19981},
}
Figure