TL;DR
Standard reinforcement learning teaches large language models to maximize a single reward, which often causes them to collapse onto a single rigid solution path. This paper introduces an automated step-level Process Reward Model (PRM) trained via Monte Carlo Tree Search (MCTS) and uses it to guide Generative Flow Networks (GFlowNets) at the reasoning-step level. The resulting models sample diverse, high-reward mathematical solutions, simultaneously improving in-domain accuracy (+2.59% on MATH Level 5) and achieving significant gains on unseen out-of-domain benchmarks (+9.4% on SAT Math).
What Problem Does This Paper Address?
In complex mathematical reasoning, arriving at the right answer through only one memorized strategy indicates brittle pattern matching rather than genuine problem-solving. While reinforcement learning from human feedback (RLHF) methods like PPO enhance accuracy, their reward-maximization objective naturally concentrates probability mass on a single "best" path—stifling solution diversity.
Furthermore, guiding multi-step reasoning requires intermediate feedback via Process Reward Models (PRMs). Traditionally, training a PRM requires costly step-by-step human annotations. Prior automated PRMs often rely on coarse binary labels and throw away intermediate rollouts, while existing GFlowNet adaptations for language models operate at the token level, disrupting semantic coherence.
Methods
The authors propose an end-to-end framework combining automated process supervision with step-level flow networks:
- Automated Continuous PRM via MCTS:
- Uses Qwen2.5-Math to perform Monte Carlo Tree Search ($k=96$ rollouts per step).
- Instead of binary labels, steps are assigned continuous values $MC(s) \in [0, 1]$ representing the empirical probability that the step leads to a correct final answer (terminating search at the first identified error).
- Rollout Reuse & Step Similarity Grouping: Rather than discarding failed or intermediate rollouts, the authors extract individual steps and cluster them based on two criteria: mathematical calculation equivalence and Levenshtein string similarity (threshold $\ge 0.85$). This expands an initial 100k steps into a rich dataset of 2.1M steps without extra compute.
- Step-Level GFlowNet Fine-Tuning:
- Instead of operating over discrete tokens, actions correspond to generating complete reasoning steps, preserving semantic meaning and mathematical validity.
- Trajectory rewards are defined multiplicatively across steps: $R(s_n) = \prod_{i=1}^n U(s_i \mid s_{i-1})$, where $U$ is the PRM score.
- Policies are trained using the Subtrajectory Balance (SubTB($\lambda$)) objective with $\lambda=1.0$ and a prioritized replay buffer. Because GFlowNets optimize to sample trajectories with probability proportional to their reward ($P(x) \propto R(x)$), the LLM learns to explore multiple distinct, high-quality reasoning pathways.
Datasets
- Training Data: 70,000 problems from OpenMathInstruct-2 for MCTS data generation; a 10,000-problem subset of challenging Level 5 problems was used for downstream policy fine-tuning.
- Evaluation Benchmarks:
- MATH Level 5 / MATH Hard (in-domain complex competition math).
- GSM8K (grade-school math word problems).
- SAT MATH (out-of-domain standardized test problems to test reasoning generalization).
- Note: No new public benchmarks were introduced, but the paper created a 2.1M-step annotated PRM training dataset via automated MCTS.
Key Insights and Ideas
- Diversity and Accuracy Are Not at Odds: PPO increases uniformity (average semantic similarity between solutions rose from 0.80 to 0.82), whereas GFlowNets decrease similarity to 0.78, proving that models can uncover alternative reasoning paths without sacrificing correctness.
- Step-Level vs. Token-Level Control: Applying GFlowNets at the step boundary matches the natural hierarchy of human mathematical proof, avoiding the semantic fragmentation that plagues token-level flow modeling.
- Generalization Through Multi-Path Exploration: Learning multiple diverse solution patterns led to dramatic generalization gains on out-of-distribution math tests, suggesting that learning multiple valid solution paths prevents overfitting to benchmark-specific heuristics.
- Unexpected Efficiency: GFlowNet fine-tuning was roughly 2× faster than the PPO baseline (1 hour vs. 2 hours on 8× A100 GPUs for 10k problems) due to faster convergence and sample efficiency from the prioritized replay buffer.
Key Results
- Accuracy Improvements:
- Llama3.2-3B-it: MATH Level 5 rose from 14.46% (base) and 15.32% (PPO) to 17.05% (+GFlowNet). On SAT MATH, accuracy leapt from 65.6% to 75.0% (+9.4% absolute gain).
- Llama3.1-8B-it: MATH Level 5 improved from 17.96% to 18.67%, and SAT MATH jumped from 81.2% to 84.4%.
- PRM Quality: Outperformed open-source alternatives like Skywork-o1-Open-PRM-Qwen-2.5-7B when used to guide test-time step search.
- Diversity Analysis: Cosine similarity across sentence embeddings (
paraphrase-MiniLM-L6-v2) dropped from 0.82 (PPO) to 0.78 (GFlowNet), yielding qualitatively different approaches (e.g., distributing division before subtraction vs. subtracting first).
Limitations and Future Work
- MCTS Upfront Overhead: Generating the PRM dataset required significant initial computation (~4 hours on 8× A100 GPUs), even though policy fine-tuning itself is fast.
- Hyperparameter Sensitivity: The framework relies on careful tuning of the similarity-grouping threshold (0.85) and replay buffer prioritization.
- Automated Data Biases: MCTS-based rollouts can occasionally propagate systematic reasoning blind spots into the PRM.
- Future Work: The authors propose extending step-level GFlowNets to offline training regimes, developing domain-specific diversity metrics for formal mathematics, and applying the method to code generation and automated theorem proving.
Summary updated