Fine-Tuning without Performance Degradation

Han Wang, Adam White, Martha WhitearXiv 2025 · DOI · arXiv

Introduction

Transitioning a reinforcement learning (RL) agent from offline training to online fine-tuning is a critical step for real-world deployment. While offline RL allows agents to learn from historical data without risky exploration, online fine-tuning is necessary to reach peak performance. However, this transition often triggers a severe "performance dip"—a sharp drop in reward as the agent begins interacting with the live environment. This degradation is primarily caused by value overestimation: the agent encounters unfamiliar states and incorrectly assumes high rewards for suboptimal actions, abandoning its safe offline policy for "phantom" gains.

The Problem: Early-Stage Degradation

In many high-stakes applications, such as hospital HVAC control or power grid management, this initial crash is unacceptable. Current solutions often struggle with either slow learning rates or the need for extensive hyperparameter tuning that is impractical in real-world settings.

Performance degradation (the "dip") observed when transitioning from offline training to online fine-tuning across various D4RL environments and data qualities.

As shown in the figure above, agents initialized with better offline policies (e.g., Expert data) often suffer the most severe relative degradation. This "Better is Worse" paradox occurs because high-performing policies have a steeper "cliff" to fall off when the agent begins exploring out-of-distribution actions with inflated value estimates.

Automatic Jump-Start (AJS)

To address this, we introduce Automatic Jump-Start (AJS), an algorithm designed to automate the transition from a stable offline policy to an aggressive online learner. AJS builds upon the Jump-Start RL (JSRL) framework, where a "guide" policy (the safe offline one) handles the initial steps of an episode before switching to an "exploration" policy (the fine-tuning one).

Key Innovations

  1. Fitted Q Evaluation (FQE): AJS utilizes off-policy estimation to calculate the real-time performance of the fine-tuned policy. This allows the system to monitor progress without relying on noisy online returns.
  2. Hybrid Policy Architecture: The system employs InAC (In-sample Actor-Critic) as a stable, degradation-resistant guide and SAC (Soft Actor-Critic) as a fast-learning exploration policy.
  3. Dynamic Scheduling: Unlike previous methods that required manual tuning of a "tolerance" hyperparameter, AJS automatically moves the "jump-start" point back only when FQE confirms the new policy is outperforming the offline baseline.

Experimental Results

We evaluated AJS using the D4RL benchmark on MuJoCo locomotion tasks (HalfCheetah, Hopper, and Walker2d). The results demonstrate that AJS significantly reduces performance degradation compared to standard SAC and other "safe" baselines like PEX and PROTO.

Comparative performance on Hopper and Walker2d Expert tasks, showing how different methods handle the offline-to-online transition.

As illustrated in the comparison above, AJS (represented by the blue line in similar experimental setups) maintains near-monotonic improvement. It reaches final performance levels comparable to the fastest online learners but bypasses the 50-80% initial performance drop typically seen in vanilla SAC. Crucially, AJS achieved these results with zero task-specific tuning, performing consistently across all tested environments.

Conclusion

AJS provides a robust "governor" for RL agents, offering the safety of conservative offline algorithms with the terminal velocity of fast online learners. While the computational cost of running real-time FQE is higher than simpler methods, the gain in stability makes it a viable candidate for industrial control problems where performance crashes carry significant real-world costs. Future work will focus on scaling AJS to high-dimensional sensory inputs and even more complex, non-stationary environments.

Summary updated

Abstract

Fine-tuning policies learned offline remains a major challenge in application domains. Monotonic performance improvement during \emph{fine-tuning} is often challenging, as agents typically experience performance degradation at the early fine-tuning stage. The community has identified multiple difficulties in fine-tuning a learned network online, however, the majority of progress has focused on improving learning efficiency during fine-tuning. In practice, this comes at a serious cost during fine-tuning: initially, agent performance degrades as the agent explores and effectively overrides the policy learned offline. We show across a range of settings, many offline-to-online algorithms exhibit either (1) performance degradation or (2) slow learning (sometimes effectively no improvement) during fine-tuning. We introduce a new fine-tuning algorithm, based on an algorithm called Jump Start, that gradually allows more exploration based on online estimates of performance. Empirically, this approach achieves fast fine-tuning and significantly reduces performance degradations compared with existing algorithms designed to do the same.
BibTeX
@misc{wang2025fine,
  author = {Han Wang and Adam White and Martha White},
  title = {{Fine-Tuning without Performance Degradation}},
  year = {2025},
  eprint = {2505.00913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00913},
}
Figure