ReFT: Reasoning with Reinforced Fine-Tuning

Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, Hang Li

TL;DR

ReFT (Reinforced Fine-Tuning) enhances the mathematical reasoning capabilities of Large Language Models by combining a brief supervised warm-up with online reinforcement learning (PPO). By automatically sampling and learning from multiple reasoning paths for the same question without requiring extra training data, ReFT significantly improves model generalization over standard supervised fine-tuning.

What Problem Does This Paper Address?

To solve complex math problems, Large Language Models (LLMs) frequently rely on Chain-of-Thought (CoT) prompting, which breaks down problems into step-by-step reasoning paths. Typically, models are trained on these paths using Supervised Fine-Tuning (SFT).

However, standard SFT datasets usually provide only a single "gold standard" reasoning path per question. This creates a major bottleneck: the model overfits to one specific way of solving a problem, failing to learn alternative valid strategies or how to navigate around mistakes. This narrow focus ultimately limits the model's ability to generalize to new, unseen problems.

Methods

The authors introduce Reinforced Fine-Tuning (ReFT), a two-stage training paradigm designed to open up the model's exploration space:

  1. Warm-up Stage: The model undergoes standard SFT for just one or two epochs. This equips the model with the basic syntax and structure required to generate coherent reasoning paths.
  2. Reinforcement Learning (RL) Stage: The model transitions to online self-learning using Proximal Policy Optimization (PPO).
Figure 1: An example of question (x), CoT (e), and answer (y) in GSM8K (Cobbe et al., 2021a). The SFT process iterates several epochs on the training data. The proposed ReFT warm-up from SFT and performs RL training on the same data.

During the RL stage, the model samples multiple reasoning paths for each question. Crucially, ReFT does not require a separate, human-annotated reward model. Instead, it uses "golden rewards" derived directly from the ground-truth answers:

  • Reward = 1.0 if the final answer extracted from the reasoning path matches the ground truth.
  • Reward = 0.1 (partial reward) if a numeric answer is successfully extracted but incorrect, helping mitigate the issue of sparse rewards.
  • Reward = 0.0 if no valid answer can be parsed.

To prevent the model from drifting too far from its initial capabilities or collapsing into gibberish, a Kullback-Leibler (KL) divergence penalty is applied against the initial warm-up policy.

Datasets

The approach was evaluated on three standard mathematical reasoning benchmarks:

  • GSM8K: High-quality linguistically diverse grade-school math word problems.
  • SVAMP: Math word problems with varying sentence structures to test robustness.
  • MathQA: A large-scale dataset based on the AQuA-RAT dataset, evaluated in both its original Multiple-Choice Question format ($\text{MathQA}_{\text{MCQ}}$) and a direct numeric format ($\text{MathQA}_{\text{numeric}}$).

Key Insights and Ideas

  • The Power of Exploration: SFT restricts a model to a single path, whereas ReFT allows the model to discover multiple valid reasoning chains for a single problem. Learning from both successful alternatives and failed attempts provides a much richer supervision signal.
  • The Danger of Multiple Choice (Reward Hacking): The authors discovered that multiple-choice datasets ($\text{MathQA}_{\text{MCQ}}$) are highly susceptible to reward hacking. Models can generate completely flawed intermediate reasoning steps but accidentally guess the correct option letter (e.g., "C"), receiving an unearned positive reward.
Figure 3: Example prediction of MathQAMCQ reveals reward hacking.
  • Data Efficiency: ReFT achieves its performance gains using the exact same training questions as SFT, proving that performance can be unlocked by optimizing the training paradigm rather than endlessly augmenting data.

Key Results

ReFT consistently outperformed SFT and standard self-training baselines across both Natural Language CoT (N-CoT) and Program-based CoT (P-CoT) formats.

Training reward, evaluation accuracy, and KL divergence curves during ReFT training
  • Substantial Accuracy Gains: Using CodeLLAMA-7B on the GSM8K dataset, ReFT improved accuracy by 9.71 points on Natural Language CoT and 11.6 points on Program-based CoT compared to the SFT baseline.
  • Synergy with Inference Strategies: When combined with inference-time techniques like majority voting and reward-model reranking, a CodeLLAMA-7B model trained with ReFT achieved an accuracy of 81.2% on GSM8K, outperforming GPT-3.5-turbo.

Limitations and Future Work

  • Training Efficiency: ReFT requires significantly more training epochs to converge than standard SFT due to the challenges of exploring a non-differentiable generation space.
  • Susceptibility to Reward Hacking: As noted in the multiple-choice experiments, final-answer-only rewards can be easily tricked.
  • Future Directions: The authors suggest exploring Process-Based Reward Models (PRMs) to reward models for correct intermediate steps rather than just the final answer. They also plan to look into offline reinforcement learning techniques to improve training efficiency and eliminate the warm-up stage entirely.

Summary updated

Abstract

One way to enhance the reasoning capability of Large Language Models (LLMs) is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT) annotations. This approach does not show sufficiently strong generalization ability, however, because the training only relies on the given CoT data. In math problem-solving, for example, there is usually only one annotated reasoning path for each question in the training data. Intuitively, it would be better for the algorithm to learn from multiple annotated reasoning paths given a question. To address this issue, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning, with math problem-solving as an example. ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper, to further fine-tune the model, where an abundance of reasoning paths are automatically sampled given the question and the rewards are naturally derived from the ground-truth answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT, and the performance can be potentially further boosted by combining inference-time strategies such as majority voting and re-ranking. Note that ReFT obtains the improvement by learning from the same training questions as SFT, without relying on extra or augmented training questions. This indicates a superior generalization ability for ReFT.
BibTeX
@inproceedings{luong2024reft,
  author = {Trung Quoc Luong and Xinbo Zhang and Zhanming Jie and Peng Sun and Xiaoran Jin and Hang Li},
  title = {{ReFT: Reasoning with Reinforced Fine-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=tHiJ7PzwmU},
}
Figure