Abstract
Offline reinforcement learning (RL) allows agents to learn from fixed datasets, but performance is often limited by the lack of exploration. Offline-to-online RL aims to bridge this by fine-tuning pre-trained models with real-time interaction. However, this transition frequently suffers from "performance collapse" and slow online improvement. We propose ENsemble-based Offline-To-Online (ENOTO) RL, a framework that utilizes Q-ensembles to stabilize the transition and accelerate learning. By loosening pessimism through a novel WeightedMinPair estimation and incorporating SUNRISE exploration, ENOTO maintains offline performance while enabling rapid online gains. Experiments on D4RL benchmarks show that ENOTO significantly improves training stability and final performance compared to existing methods.
Introduction
The transition from offline pre-training to online fine-tuning is a critical challenge in reinforcement learning. While offline RL provides a safe starting point, the shift to online interaction often triggers a sharp drop in performance—a phenomenon known as "performance collapse." This occurs because the agent must shift from a strictly pessimistic evaluation (to avoid out-of-distribution actions) to an optimistic one (to explore). Existing methods often struggle to balance these conflicting requirements, leading to either instability or stagnant improvement.
The ENOTO Framework
ENOTO addresses these challenges through three integrated components designed to bridge the gap between offline and online learning phases:
1. Robust Q-Ensembles
Standard RL agents typically use two Q-networks. ENOTO scales this to an ensemble of $N$ networks (e.g., $N=10$). This larger ensemble provides a more reliable estimate of uncertainty and acts as an implicit stabilizer during the transition phase, preventing the agent from "forgetting" its offline-learned skills.
2. Strategic Loosening of Pessimism (WeightedMinPair)
In the offline phase, strict pessimism is necessary to avoid overestimating unknown actions. However, maintaining this online prevents exploration. ENOTO introduces WeightedMinPair, which calculates the expectation over random pairs of Q-networks. This approach "loosens" the pessimism just enough to encourage exploration without falling into the trap of chasing "fake" high-value actions.
3. Ensemble-Based Exploration (SUNRISE)
To further boost sample efficiency, ENOTO integrates the SUNRISE mechanism. By measuring the variance across the Q-ensemble, the agent can identify high-uncertainty regions of the environment and prioritize them for exploration, accelerating the discovery of superior policies.
Experimental Evaluation
We evaluated ENOTO on the D4RL suite, focusing on MuJoCo locomotion and Antmaze navigation tasks. The framework was tested as a "plugin" for existing algorithms like CQL and LAPO.
Stability and Performance
As shown in Figure 1(a), standard transitions (e.g., CQL to SAC) often result in a significant initial performance drop. In contrast, ENOTO (represented by CQL-N -> SAC-N) maintains or immediately improves upon its offline performance. This stability is attributed to the implicit pessimism provided by the larger Q-ensemble, which keeps the agent's actions closer to the known data distribution initially.
Component Analysis
Ablation studies highlight the importance of each component. Figure 3 demonstrates that while the WeightedMinPair base provides a strong foundation for online improvement, the addition of exploration mechanisms like SUNRISE leads to the highest final performance and fastest learning rates.
Key Insights
- Ensembles as an Implicit Bridge: Increasing the number of Q-networks provides a natural stabilizer during the transition phase, preventing catastrophic performance drops even when explicit offline penalties are removed.
- Pessimism is a Spectrum: The transition from offline to online is not a binary switch. A gradual loosening of pessimism via WeightedMinPair is essential for efficient fine-tuning.
- Data Efficiency: In the online phase, focusing on new interactions rather than being anchored to old offline data can improve sample efficiency when using an ensemble, as it prevents the agent from being stuck in suboptimal behaviors present in the original dataset.
Conclusion
ENOTO provides a versatile and effective framework for offline-to-online RL. By leveraging Q-ensembles and strategic pessimism management, it solves the performance collapse problem and accelerates online improvement. While the computational cost of ensembles is a consideration, the gains in stability and performance make it a compelling solution for real-world RL applications where pre-training on historical data is common.
Summary updated