TL;DR
This paper introduces Reinforced Fine-Tuning (ReFT), a novel method to improve LLM reasoning by leveraging reinforcement learning to explore multiple reasoning paths, rather than relying on a single supervised path. ReFT significantly outperforms standard fine-tuning methods on math reasoning tasks, demonstrating enhanced generalization capabilities.
What Problem Does This Paper Address?
Large Language Models (LLMs) often struggle with complex reasoning tasks, even when fine-tuned using Chain-of-Thought (CoT) annotations. The standard Supervised Fine-Tuning (SFT) approach typically uses only one pre-defined reasoning path per problem. This limited exposure can hinder the model's ability to generalize, as there are often multiple valid ways to arrive at a correct answer. The authors argue that learning from a diverse set of reasoning paths is crucial for improving generalization, especially in domains like math problem-solving where such diversity is common.
Methods
The core of ReFT is a two-stage process:
- Warm-up Stage (SFT): The model is initially fine-tuned using standard Supervised Fine-Tuning (SFT) on existing (question, CoT, answer) data. This stage equips the model with basic problem-solving skills.
- Reinforcement Learning Stage (RL): After the warm-up, the model is further fine-tuned using an online reinforcement learning algorithm, specifically Proximal Policy Optimization (PPO). In this stage, the model actively samples multiple CoT reasoning paths for a given question. The reward signal is derived directly from the ground-truth answer, eliminating the need for a separate reward model. This allows the model to learn from a richer set of reasoning strategies and improve its generalization.
Datasets
The authors evaluated ReFT on three standard math problem-solving datasets:
- GSM8K: A dataset of grade-school math problems.
- SVAMP: A dataset of math word problems designed to be challenging.
- MathQA: A dataset containing math problems with multiple-choice answers.
They also used a numeric version of MathQA to investigate reward hacking. The CoT annotations (both natural language and program-based) were generated using GPT-3.5-turbo.
Key Insights and Ideas
- Diversity is Key: Relying on a single CoT path for SFT limits generalization. ReFT's strength lies in its ability to explore and learn from multiple valid reasoning paths.
- No Extra Data Needed: ReFT achieves significant improvements without requiring additional or augmented training data, making it a practical approach.
- Natural Rewards: Deriving rewards directly from ground-truth answers simplifies the RL setup and avoids the complexity of training separate reward models, as seen in methods like RLHF.
- Reward Hacking Vulnerability: For multiple-choice questions, the reward function can be easily manipulated if the model learns to output the correct choice regardless of the reasoning path's validity, a phenomenon observed in the MathQA dataset.
Key Results
- Significant Performance Gains: ReFT consistently outperformed SFT across all tested datasets, achieving up to a 10-point improvement on GSM8K N-CoT and P-CoT with CodeLLAMA. On average, ReFT provided 6.7 and 7.4 point improvements over SFT with CodeLLAMA on all datasets for N-CoT and P-CoT respectively.
- Robustness: ReFT showed strong performance across different foundation models (Galactica and CodeLLAMA) and both natural language and program-based CoTs.
- Compatibility: ReFT's performance can be further boosted by common inference-time strategies like majority voting and reward model reranking.
- Competitive with State-of-the-Art: The best ReFT variant, when combined with reranking, surpassed GPT-3.5-turbo on GSM8K, despite using a smaller model.
Limitations and Future Work
- Training Efficiency: ReFT requires more training epochs to converge compared to SFT, as it explores a larger search space. This can lead to increased computational costs.
- Reward Hacking: The reliance on final answer-based rewards can lead to reward hacking, particularly in multiple-choice settings. The authors suggest that more detailed, process-based reward functions could mitigate this.
- Future Work: The authors propose exploring offline reinforcement learning techniques, developing warm-up-free methods, and investigating process-based rewards for more robust training. They also aim to apply ReFT to other general reasoning tasks beyond math.
Summary updated