Introduction
Reinforcement learning (RL) is a powerful framework for aligning Large Language Models (LLMs) with complex, long-term objectives. However, "online" RL—where models learn through real-time interaction—presents significant hurdles in production environments due to high computational costs and potential safety risks. This has catalyzed interest in offline RL, which leverages pre-existing datasets of interactions.
Despite its promise, current offline RL techniques like Direct Preference Optimization (DPO) or Proximal Policy Optimization (PPO) often struggle with numerical instability. These methods frequently rely on propensity score ratios that can lead to exploding gradients when the model's policy diverges from the training data. Researchers at Adobe have addressed these challenges by introducing Refit and Swift, two algorithms that frame offline RL as a stable, reward-weighted variant of supervised fine-tuning (SFT).
Methodology: Simplifying the RL Objective
The core innovation of this work lies in simplifying the mathematical machinery required for RL by treating it as a weighted version of standard supervised fine-tuning.
Refit (Reward-Weighted Fine-Tuning)
The authors demonstrate that maximizing a lower bound of the standard online RL objective is mathematically equivalent to performing SFT where each training trajectory is weighted by its total reward. By bypassing token-level ratios, Refit maintains the stability of standard fine-tuning while effectively optimizing for cumulative rewards.
Swift: Enhancing Efficiency through Standardization
Swift builds upon Refit by introducing reward standardization. By centering rewards (subtracting the mean and dividing by the standard deviation) for specific contexts, Swift reduces gradient variance. This allows the model to learn more effectively from the relative quality of different responses to the same prompt, rather than relying on absolute reward values.
Crucially, both methods utilize a "global" reward attribution strategy, where the total trajectory reward is applied equally to every token in the sequence. This eliminates the need for complex, token-level reward models.
Experimental Setup
The algorithms were evaluated across six diverse benchmarks, including reasoning tasks (ARC, OpenBookQA, SciQA, MMLU) and specialized conversational tasks (CoSQL, MathDial). The models were tested in two distinct modes: Reasoning (requiring the agent to think through a problem step-by-step) and Clarifying Questions (requiring the agent to identify missing information before answering).
Results and Analysis
The experimental results demonstrate that Swift and Refit consistently outperform traditional SFT baselines and more complex methods like StepDPO.
Accuracy and Quality Gains
Swift emerged as the top performer in 7 out of 12 experimental settings and was among the top two in 10 out of 12. Beyond raw accuracy, the models showed significant improvements in conversational quality. Using GPT-4o as a judge, the researchers evaluated responses across several dimensions: reasoning, comprehensiveness, and pedagogical value.
Key Insights
- Direct Optimization: Unlike DPO, which converts rewards into preferences, Refit and Swift optimize rewards directly. This reduces information loss and leads to superior final performance.
- Stability via Simplicity: By avoiding propensity ratios, the authors eliminate the "exploding gradient" problems that often plague offline PPO implementations.
- The Difficulty of Clarification: The models achieved higher accuracy gains in reasoning tasks than in clarification tasks. This suggests that the policy for when to ask a question is more difficult for LLMs to master than the logic required to solve a known problem.
Conclusion and Future Work
Refit and Swift provide a robust and scalable path for offline RL in LLMs. By treating RL as a weighted version of SFT, these methods achieve high performance without the instability of traditional approaches. While current evaluations rely on LLM-based judges and specific QA datasets, future work aims to explore these algorithms in more subjective domains like creative writing and complex software engineering tasks.
Summary updated