TL;DR
ReFT (Reinforced Fine-Tuning) enhances the mathematical reasoning capabilities of Large Language Models by combining a brief supervised warm-up with online reinforcement learning (PPO). By automatically sampling and learning from multiple reasoning paths for the same question without requiring extra training data, ReFT significantly improves model generalization over standard supervised fine-tuning.
What Problem Does This Paper Address?
To solve complex math problems, Large Language Models (LLMs) frequently rely on Chain-of-Thought (CoT) prompting, which breaks down problems into step-by-step reasoning paths. Typically, models are trained on these paths using Supervised Fine-Tuning (SFT).
However, standard SFT datasets usually provide only a single "gold standard" reasoning path per question. This creates a major bottleneck: the model overfits to one specific way of solving a problem, failing to learn alternative valid strategies or how to navigate around mistakes. This narrow focus ultimately limits the model's ability to generalize to new, unseen problems.
Methods
The authors introduce Reinforced Fine-Tuning (ReFT), a two-stage training paradigm designed to open up the model's exploration space:
- Warm-up Stage: The model undergoes standard SFT for just one or two epochs. This equips the model with the basic syntax and structure required to generate coherent reasoning paths.
- Reinforcement Learning (RL) Stage: The model transitions to online self-learning using Proximal Policy Optimization (PPO).
During the RL stage, the model samples multiple reasoning paths for each question. Crucially, ReFT does not require a separate, human-annotated reward model. Instead, it uses "golden rewards" derived directly from the ground-truth answers:
- Reward = 1.0 if the final answer extracted from the reasoning path matches the ground truth.
- Reward = 0.1 (partial reward) if a numeric answer is successfully extracted but incorrect, helping mitigate the issue of sparse rewards.
- Reward = 0.0 if no valid answer can be parsed.
To prevent the model from drifting too far from its initial capabilities or collapsing into gibberish, a Kullback-Leibler (KL) divergence penalty is applied against the initial warm-up policy.
Datasets
The approach was evaluated on three standard mathematical reasoning benchmarks:
- GSM8K: High-quality linguistically diverse grade-school math word problems.
- SVAMP: Math word problems with varying sentence structures to test robustness.
- MathQA: A large-scale dataset based on the AQuA-RAT dataset, evaluated in both its original Multiple-Choice Question format ($\text{MathQA}_{\text{MCQ}}$) and a direct numeric format ($\text{MathQA}_{\text{numeric}}$).
Key Insights and Ideas
- The Power of Exploration: SFT restricts a model to a single path, whereas ReFT allows the model to discover multiple valid reasoning chains for a single problem. Learning from both successful alternatives and failed attempts provides a much richer supervision signal.
- The Danger of Multiple Choice (Reward Hacking): The authors discovered that multiple-choice datasets ($\text{MathQA}_{\text{MCQ}}$) are highly susceptible to reward hacking. Models can generate completely flawed intermediate reasoning steps but accidentally guess the correct option letter (e.g., "C"), receiving an unearned positive reward.
- Data Efficiency: ReFT achieves its performance gains using the exact same training questions as SFT, proving that performance can be unlocked by optimizing the training paradigm rather than endlessly augmenting data.
Key Results
ReFT consistently outperformed SFT and standard self-training baselines across both Natural Language CoT (N-CoT) and Program-based CoT (P-CoT) formats.
- Substantial Accuracy Gains: Using
CodeLLAMA-7Bon the GSM8K dataset, ReFT improved accuracy by 9.71 points on Natural Language CoT and 11.6 points on Program-based CoT compared to the SFT baseline. - Synergy with Inference Strategies: When combined with inference-time techniques like majority voting and reward-model reranking, a
CodeLLAMA-7Bmodel trained with ReFT achieved an accuracy of 81.2% on GSM8K, outperformingGPT-3.5-turbo.
Limitations and Future Work
- Training Efficiency: ReFT requires significantly more training epochs to converge than standard SFT due to the challenges of exploring a non-differentiable generation space.
- Susceptibility to Reward Hacking: As noted in the multiple-choice experiments, final-answer-only rewards can be easily tricked.
- Future Directions: The authors suggest exploring Process-Based Reward Models (PRMs) to reward models for correct intermediate steps rather than just the final answer. They also plan to look into offline reinforcement learning techniques to improve training efficiency and eliminate the warm-up stage entirely.
Summary updated