Abstract: Offline reinforcement learning (RL) is a learning paradigm where an agent learns from a fixed dataset of experience. However, learning solely from a static dataset can limit the performance due to the lack of exploration. To overcome it, offline-to-online RL combines offline pre-training with online fine-tuning, which enables the agent to further refine its policy by interacting with the environment in real-time. Despite its benefits, existing offline-to-online RL methods suffer from performance degradation and slow improvement during the online phase. To tackle these challenges, we propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Moreover, to expedite online performance enhancement, we appropriately loosen the pessimism of Q-value estimation and incorporate ensemble-based exploration mechanisms into our framework. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods during online fine-tuning on a range of locomotion and navigation tasks, significantly outperforming existing offline-to-online RL methods.
The Challenge of the Offline-to-Online Transition
Reinforcement Learning typically operates in two distinct modes: Offline RL, which learns safely from fixed datasets, and Online RL, which learns through active environmental interaction. The ideal workflow is Offline-to-Online RL, where an agent is pre-trained on a dataset and then fine-tuned in the real world.
However, this transition is notoriously difficult. Offline agents are inherently pessimistic to avoid overestimating the value of unseen actions. When transitioning to online learning, this pessimism creates a dilemma:
- Performance Collapse: Removing pessimistic constraints suddenly can cause value estimates to fluctuate wildly, leading the agent to "forget" its offline knowledge.
- Stagnation: Maintaining strict pessimism prevents the agent from exploring new, potentially superior actions, leading to slow or non-existent improvement.
The ENOTO Framework
To solve these issues, we propose ENOTO, a framework that utilizes a large ensemble of Q-networks (typically 10) to manage the transition through three key stages:
- Q-Ensemble Integration: During offline pre-training, the agent uses an ensemble of $N$ Q-networks. It enforces a strict pessimistic bound by taking the minimum of all networks, ensuring behavior stays within the dataset's distribution.
- Loosening Pessimism (WeightedMinPair): Upon moving online, ENOTO "loosens the reins" using WeightedMinPair. Instead of a global minimum, it computes the expectation of minimums from random pairs of Q-functions. This provides a "softer" pessimism that maintains stability without stifling the agent's ability to learn from new data.
- Optimistic Exploration (SUNRISE): To accelerate learning, we incorporate the SUNRISE mechanism. This uses the variance across the Q-ensemble to identify uncertainty and encourage the agent to explore states it is less confident about, shifting the mindset from avoidance to active discovery.
Key Results and Stability
Evaluations on the D4RL benchmark, including MuJoCo locomotion and complex Antmaze navigation tasks, show that ENOTO effectively bridges the performance gap.
As illustrated in Figure 2, standard methods often experience a sharp dip in normalized return and volatile Q-value estimates during the phase switch. ENOTO-based methods (e.g., CQL-N to SAC-N) maintain steady performance and stable value estimates, acting as a "stability bridge" between the two learning paradigms.
Driving Performance through Exploration
The integration of ensemble-based exploration is critical for overcoming the stagnation seen in traditional fine-tuning. By strategically dialing down conservativeness, ENOTO-trained agents explore regions of the state space that standard offline agents never reach.
Figure 6 highlights how the SUNRISE mechanism, when paired with our WeightedMinPair approach, significantly outperforms other exploration strategies. This synergy allows the agent to improve rapidly even in "sparse reward" environments like Antmaze, where signals are infrequent.
Conclusion and Future Work
ENOTO demonstrates that Q-ensembles are a powerful tool for balancing the competing needs of pessimism and exploration. While the computational overhead of running 10 Q-networks is a consideration, the gains in stability and efficiency are substantial. Future research will explore "ensemble distillation" to compress this knowledge into smaller, more deployable models for real-world robotics.
Summary updated