ReFT: Reasoning with Reinforced Fine-Tuning

Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, Hang Li

TL;DR

This paper introduces Reinforced Fine-Tuning (ReFT), a novel method to improve LLM reasoning by leveraging reinforcement learning to explore multiple reasoning paths, rather than relying on a single supervised path. ReFT significantly outperforms standard fine-tuning methods on math reasoning tasks, demonstrating enhanced generalization capabilities.

What Problem Does This Paper Address?

Large Language Models (LLMs) often struggle with complex reasoning tasks, even when fine-tuned using Chain-of-Thought (CoT) annotations. The standard Supervised Fine-Tuning (SFT) approach typically uses only one pre-defined reasoning path per problem. This limited exposure can hinder the model's ability to generalize, as there are often multiple valid ways to arrive at a correct answer. The authors argue that learning from a diverse set of reasoning paths is crucial for improving generalization, especially in domains like math problem-solving where such diversity is common.

Methods

The core of ReFT is a two-stage process:

  1. Warm-up Stage (SFT): The model is initially fine-tuned using standard Supervised Fine-Tuning (SFT) on existing (question, CoT, answer) data. This stage equips the model with basic problem-solving skills.
  2. Reinforcement Learning Stage (RL): After the warm-up, the model is further fine-tuned using an online reinforcement learning algorithm, specifically Proximal Policy Optimization (PPO). In this stage, the model actively samples multiple CoT reasoning paths for a given question. The reward signal is derived directly from the ground-truth answer, eliminating the need for a separate reward model. This allows the model to learn from a richer set of reasoning strategies and improve its generalization.

Datasets

The authors evaluated ReFT on three standard math problem-solving datasets:

  • GSM8K: A dataset of grade-school math problems.
  • SVAMP: A dataset of math word problems designed to be challenging.
  • MathQA: A dataset containing math problems with multiple-choice answers.

They also used a numeric version of MathQA to investigate reward hacking. The CoT annotations (both natural language and program-based) were generated using GPT-3.5-turbo.

Key Insights and Ideas

  • Diversity is Key: Relying on a single CoT path for SFT limits generalization. ReFT's strength lies in its ability to explore and learn from multiple valid reasoning paths.
  • No Extra Data Needed: ReFT achieves significant improvements without requiring additional or augmented training data, making it a practical approach.
  • Natural Rewards: Deriving rewards directly from ground-truth answers simplifies the RL setup and avoids the complexity of training separate reward models, as seen in methods like RLHF.
  • Reward Hacking Vulnerability: For multiple-choice questions, the reward function can be easily manipulated if the model learns to output the correct choice regardless of the reasoning path's validity, a phenomenon observed in the MathQA dataset.

Key Results

  • Significant Performance Gains: ReFT consistently outperformed SFT across all tested datasets, achieving up to a 10-point improvement on GSM8K N-CoT and P-CoT with CodeLLAMA. On average, ReFT provided 6.7 and 7.4 point improvements over SFT with CodeLLAMA on all datasets for N-CoT and P-CoT respectively.
  • Robustness: ReFT showed strong performance across different foundation models (Galactica and CodeLLAMA) and both natural language and program-based CoTs.
  • Compatibility: ReFT's performance can be further boosted by common inference-time strategies like majority voting and reward model reranking.
  • Competitive with State-of-the-Art: The best ReFT variant, when combined with reranking, surpassed GPT-3.5-turbo on GSM8K, despite using a smaller model.

Limitations and Future Work

  • Training Efficiency: ReFT requires more training epochs to converge compared to SFT, as it explores a larger search space. This can lead to increased computational costs.
  • Reward Hacking: The reliance on final answer-based rewards can lead to reward hacking, particularly in multiple-choice settings. The authors suggest that more detailed, process-based reward functions could mitigate this.
  • Future Work: The authors propose exploring offline reinforcement learning techniques, developing warm-up-free methods, and investigating process-based rewards for more robust training. They also aim to apply ReFT to other general reasoning tasks beyond math.

Summary updated

Abstract

One way to enhance the reasoning capability of Large Language Models (LLMs) is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT) annotations. This approach does not show sufficiently strong generalization ability, however, because the training only relies on the given CoT data. In math problem-solving, for example, there is usually only one annotated reasoning path for each question in the training data. Intuitively, it would be better for the algorithm to learn from multiple annotated reasoning paths given a question. To address this issue, we propose a simple yet effective approach called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of learning LLMs for reasoning, with math problem-solving as an example. ReFT first warmups the model with SFT, and then employs on-line reinforcement learning, specifically the PPO algorithm in this paper, to further fine-tune the model, where an abundance of reasoning paths are automatically sampled given the question and the rewards are naturally derived from the ground-truth answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that ReFT significantly outperforms SFT, and the performance can be potentially further boosted by combining inference-time strategies such as majority voting and re-ranking. Note that ReFT obtains the improvement by learning from the same training questions as SFT, without relying on extra or augmented training questions. This indicates a superior generalization ability for ReFT.
BibTeX
@inproceedings{luong2024reft,
  author = {Trung Quoc Luong and Xinbo Zhang and Zhanming Jie and Peng Sun and Xiaoran Jin and Hang Li},
  title = {{ReFT: Reasoning with Reinforced Fine-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=tHiJ7PzwmU},
}
Figure