ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng

Abstract

Transitioning reinforcement learning (RL) agents from static offline datasets to real-time online interaction remains a significant challenge. Standard approaches often suffer from a catastrophic performance "crash" during the initial fine-tuning phase due to distributional shifts and the abrupt removal of offline pessimism. We introduce ENOTO (ENsemble-based Offline-To-Online), a framework that utilizes Q-ensembles to provide a flexible constraint during this transition. By employing a novel WeightedMinPair target computation, ENOTO balances the need for stability with the optimism required for exploration. Our evaluations on the D4RL benchmark suite demonstrate that ENOTO consistently prevents performance degradation and significantly accelerates policy improvement compared to existing state-of-the-art methods.

Introduction

Reinforcement learning agents are increasingly trained in two stages: an offline phase, where they learn from a fixed dataset, followed by an online fine-tuning phase, where they interact with the live environment to improve performance. However, this transition is fraught with two major hurdles:

  1. Performance Degradation: Naively switching from an offline method to an online one often causes a catastrophic drop in performance. The "distributional shift" between the offline data and new online experiences creates massive initial errors in the agent's value estimates, leading to the "forgetting" of offline-learned behaviors.
  2. Slow Improvement: Offline RL methods are intentionally pessimistic to avoid overestimating values for actions not in the dataset. Maintaining this strict pessimism during the online phase stifles the exploration necessary to surpass the performance of the initial dataset.
Figure 1: The performance crash observed when naively switching from CQL to SAC, and the stability provided by the ENOTO approach (CQL-N -> SAC-N).

The ENOTO Framework

ENOTO addresses these challenges by replacing standard Q-networks with an ensemble of $N$ networks. This ensemble architecture serves as a versatile constraint that manages the transition across phases more effectively than traditional explicit penalty terms.

Loosening Pessimism with WeightedMinPair

A core innovation of ENOTO is the WeightedMinPair target computation used during online fine-tuning. Instead of taking the absolute minimum of all $N$ networks—which is often too conservative—or the mean—which can be unstable—WeightedMinPair computes the expected minimum over random pairs of Q-functions. This provides a "Goldilocks" level of conservatism: stable enough to prevent the initial crash but optimistic enough to allow for policy growth.

Optimistic Exploration

To further accelerate learning, ENOTO integrates uncertainty-based exploration. By utilizing the variance across the Q-ensemble, the agent can weight Bellman backups and prioritize exploration in unknown regions of the state-action space, effectively overcoming the limitations of static offline policies.

Key Insights and Results

Our experiments on the D4RL benchmark suite, including MuJoCo locomotion and complex Antmaze navigation tasks, reveal several key insights:

  • Ensembles as Buffer: Increasing the number of Q-networks provides a natural buffer against distributional shifts. The initial "crash" is largely caused by an "oblivion of information" due to large temporal difference (TD) errors when pessimism is removed too abruptly.
  • Target Computation Matters: Ablation studies show that WeightedMinPair significantly outperforms other common ensemble targets like MinQ, MeanQ, or REM in terms of both stability and final return.
Figure 2: Comparison of different Q-target computation methods, highlighting the superior stability and efficiency of the WeightedMinPair approach.

ENOTO consistently avoids the performance drops seen in other state-of-the-art methods like PEX and BR. On difficult navigation tasks like Antmaze, ENOTO demonstrates superior ability to handle sparse rewards and complex multi-step reasoning through its balanced exploration strategy.

Conclusion and Future Work

ENOTO provides a robust and efficient framework for the critical transition from offline to online reinforcement learning. While the use of ensembles increases computational overhead, the gains in stability and convergence speed are substantial. Future research will focus on mitigating computational costs through techniques like ensemble distillation and testing the framework on high-dimensional, real-world robotic tasks.

Summary updated

BibTeX
@inproceedings{zhao2024enoto,
  author = {Kai Zhao and Jianye Hao and Yi Ma and Jinyi Liu and Yan Zheng},
  title = {{ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure