ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

Kai Zhao, Yi Ma, Jinyi Liu, Zheng Yan, Zhaopeng Meng, Meng, Zhaopeng

Abstract

Offline reinforcement learning (RL) allows agents to learn from fixed datasets, but performance is often limited by the lack of exploration. Offline-to-online RL aims to bridge this by fine-tuning pre-trained models with real-time interaction. However, this transition frequently suffers from "performance collapse" and slow online improvement. We propose ENsemble-based Offline-To-Online (ENOTO) RL, a framework that utilizes Q-ensembles to stabilize the transition and accelerate learning. By loosening pessimism through a novel WeightedMinPair estimation and incorporating SUNRISE exploration, ENOTO maintains offline performance while enabling rapid online gains. Experiments on D4RL benchmarks show that ENOTO significantly improves training stability and final performance compared to existing methods.

Introduction

The transition from offline pre-training to online fine-tuning is a critical challenge in reinforcement learning. While offline RL provides a safe starting point, the shift to online interaction often triggers a sharp drop in performance—a phenomenon known as "performance collapse." This occurs because the agent must shift from a strictly pessimistic evaluation (to avoid out-of-distribution actions) to an optimistic one (to explore). Existing methods often struggle to balance these conflicting requirements, leading to either instability or stagnant improvement.

The ENOTO Framework

ENOTO addresses these challenges through three integrated components designed to bridge the gap between offline and online learning phases:

1. Robust Q-Ensembles

Standard RL agents typically use two Q-networks. ENOTO scales this to an ensemble of $N$ networks (e.g., $N=10$). This larger ensemble provides a more reliable estimate of uncertainty and acts as an implicit stabilizer during the transition phase, preventing the agent from "forgetting" its offline-learned skills.

2. Strategic Loosening of Pessimism (WeightedMinPair)

In the offline phase, strict pessimism is necessary to avoid overestimating unknown actions. However, maintaining this online prevents exploration. ENOTO introduces WeightedMinPair, which calculates the expectation over random pairs of Q-networks. This approach "loosens" the pessimism just enough to encourage exploration without falling into the trap of chasing "fake" high-value actions.

3. Ensemble-Based Exploration (SUNRISE)

To further boost sample efficiency, ENOTO integrates the SUNRISE mechanism. By measuring the variance across the Q-ensemble, the agent can identify high-uncertainty regions of the environment and prioritize them for exploration, accelerating the discovery of superior policies.

Figure 1(a): Normalized return during the offline-to-online transition. ENOTO (CQL-N -> SAC-N) avoids the performance collapse seen in standard transitions (CQL -> SAC).

Experimental Evaluation

We evaluated ENOTO on the D4RL suite, focusing on MuJoCo locomotion and Antmaze navigation tasks. The framework was tested as a "plugin" for existing algorithms like CQL and LAPO.

Stability and Performance

As shown in Figure 1(a), standard transitions (e.g., CQL to SAC) often result in a significant initial performance drop. In contrast, ENOTO (represented by CQL-N -> SAC-N) maintains or immediately improves upon its offline performance. This stability is attributed to the implicit pessimism provided by the larger Q-ensemble, which keeps the agent's actions closer to the known data distribution initially.

Component Analysis

Ablation studies highlight the importance of each component. Figure 3 demonstrates that while the WeightedMinPair base provides a strong foundation for online improvement, the addition of exploration mechanisms like SUNRISE leads to the highest final performance and fastest learning rates.

Figure 3: Impact of different components on normalized return. The combination of WeightedMinPair and SUNRISE yields the best results for online fine-tuning.

Key Insights

  • Ensembles as an Implicit Bridge: Increasing the number of Q-networks provides a natural stabilizer during the transition phase, preventing catastrophic performance drops even when explicit offline penalties are removed.
  • Pessimism is a Spectrum: The transition from offline to online is not a binary switch. A gradual loosening of pessimism via WeightedMinPair is essential for efficient fine-tuning.
  • Data Efficiency: In the online phase, focusing on new interactions rather than being anchored to old offline data can improve sample efficiency when using an ensemble, as it prevents the agent from being stuck in suboptimal behaviors present in the original dataset.

Conclusion

ENOTO provides a versatile and effective framework for offline-to-online RL. By leveraging Q-ensembles and strategic pessimism management, it solves the performance collapse problem and accelerates online improvement. While the computational cost of ensembles is a consideration, the gains in stability and performance make it a compelling solution for real-world RL applications where pre-training on historical data is common.

Summary updated

Abstract

Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.
BibTeX
@inproceedings{zhao2024enoto,
  author = {Kai Zhao and Yi Ma and Jinyi Liu and Zheng Yan and Zhaopeng Meng and Meng, Zhaopeng},
  title = {{ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure