Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble

Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, Jinwoo Shin

TL;DR

This paper tackles the challenge of fine-tuning robots trained offline using new online data. They introduce a "balanced replay" strategy to smartly mix old and new data and a "pessimistic Q-ensemble" to prevent the robot from making dangerous mistakes early on. This approach leads to more efficient and robust learning for robots in real-world tasks.

What Problem Does This Paper Address?

Deep offline reinforcement learning (RL) has made impressive strides, allowing us to train capable robotic agents solely from pre-collected datasets. However, in many practical scenarios, it's beneficial to further refine these agents with a bit of real-world interaction (online learning). The core problem here is "distribution shift." When an agent trained offline encounters new situations or actions online that weren't well-represented in its training data, its learned value functions (Q-functions) can become highly inaccurate. This leads to "bootstrap errors," where bad predictions propagate and can quickly degrade the agent's performance, essentially "destroying" the good policy learned offline. This paper aims to bridge the gap between offline and online RL by enabling safe and efficient fine-tuning without catastrophic performance drops.

Methods

The authors propose a two-pronged approach to address the distribution shift problem:

  1. Balanced Replay Scheme: Instead of uniformly sampling from a combined offline and online dataset, this method intelligently prioritizes samples. It uses a learned "online-ness" measure to favor samples that are closer to the current online policy while still incorporating relevant, near-on-policy samples from the offline dataset. This helps the agent learn more efficiently by focusing on useful data.
  2. Pessimistic Q-Ensemble Scheme: To combat over-optimism about unfamiliar actions in novel states, the authors leverage an ensemble of multiple Q-functions. These Q-functions are initially trained pessimistically (e.g., using Conservative Q-Learning - CQL) offline. This pessimism acts as a safeguard, preventing the agent from making potentially dangerous choices early in the fine-tuning process. As the agent gathers more online data, the ensemble's pessimism gradually fades, allowing for more exploration.

Datasets

The paper evaluates their method on several standard benchmarks:

  • MuJoCo Locomotion Tasks: These include halfcheetah, hopper, and walker2d from the D4RL benchmark suite. They use four types of datasets for these tasks: random, medium, medium-replay, and medium-expert, representing varying levels of data quality and coverage.
  • Vision-based Robotic Manipulation Tasks: These are sparse-reward, pixel-based tasks from Singh et al. [10], including pick-place, grasp-closed-drawer, and grasp-blocked-drawer. For these tasks, they create a more realistic setup by replacing half of the original structured dataset with uniform random policy rollouts.

Key Insights and Ideas

  • Distribution Shift is the Enemy: The paper clearly demonstrates that naive fine-tuning with online data can be disastrous due to distribution shift, leading to severe performance degradation.
  • Offline Data is Still Valuable: Even when fine-tuning online, strategically using relevant offline data can significantly improve sample efficiency and final performance.
  • Pessimism as a Safety Net: Initial pessimism in Q-functions is crucial for stabilizing early online learning. It prevents the agent from exploiting potentially bad actions in unseen states.
  • Ensembles for Robustness: Using an ensemble of pessimistic Q-functions provides a more robust estimate of value, making the agent less susceptible to individual Q-function errors.
  • Balanced Replay is Key: The proposed balanced replay scheme is shown to be effective in selecting useful offline samples, adapting to the agent's evolving policy.

Key Results

  • The proposed method (Offline-to-Online RL via Balanced Replay and Pessimistic Q-Ensemble) consistently outperforms baseline methods across various locomotion and manipulation tasks in terms of both sample efficiency and final performance.
  • In experiments, the method significantly outperforms baselines like SAC-ft (fine-tuning a CQL-initialized agent with SAC), highlighting the necessity of both balanced replay and the pessimistic Q-ensemble.
  • The balanced replay scheme effectively utilizes offline samples only when they are relevant to the current policy, as shown by buffer analysis.
  • The pessimistic Q-ensemble demonstrates superior discriminative ability in distinguishing seen from unseen state-action pairs, leading to more stable fine-tuning.
  • Ablation studies show that both balanced replay and ensemble size contribute to improved performance, with larger ensembles generally yielding better results up to a point.

Limitations and Future Work

  • Computational Overhead: The ensemble approach incurs additional computational cost for training and inference, although parallelization can mitigate this.
  • Dataset Composition: The paper notes that dataset composition can influence fine-tuning performance in counterintuitive ways, with more diverse datasets sometimes leading to better asymptotic performance.
  • Future Work: The authors suggest that their method could be useful for other related topics like scalable RL and RL safety. They also mention exploring alternative density ratio schemes, which showed similar performance but might have potential benefits for asymptotic trending.

Summary updated

Abstract

Recent advance in deep offline reinforcement learning (RL) has made it possible to train strong robotic agents from offline datasets. However, depending on the quality of the trained agents and the application being considered, it is often desirable to fine-tune such agents via further online interactions. In this paper, we observe that state-action distribution shift may lead to severe bootstrap error during fine-tuning, which destroys the good initial policy obtained via offline RL. To address this issue, we first propose a balanced replay scheme that prioritizes samples encountered online while also encouraging the use of near-on-policy samples from the offline dataset. Furthermore, we leverage multiple Q-functions trained pessimistically offline, thereby preventing overoptimism concerning unfamiliar actions at novel states during the initial training phase. We show that the proposed method improves sample-efficiency and final performance of the fine-tuned robotic agents on various locomotion and manipulation tasks. Our code is available at: https://github.com/shlee94/Off2OnRL.
BibTeX
@inproceedings{lee2021offline,
  author = {Seunghyun Lee and Younggyo Seo and Kimin Lee and Pieter Abbeel and Jinwoo Shin},
  title = {{Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble}},
  booktitle = {Conference on Robot Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=reWvHCBzGq},
}
Figure