Offline Reinforcement Learning with Imbalanced Datasets

Li Jiang, Sijie Chen, Jielin Qiu, Haoran Xu, Wai Kin Chan, Zhao DingarXiv 2023 · DOI · arXiv

Abstract

Standard offline reinforcement learning (RL) algorithms often struggle with real-world datasets where state coverage is heavily imbalanced, typically following a power-law distribution. This paper introduces Retrieval-Based Conservative Q-Learning (RB-CQL), which augments agents with a non-parametric retrieval process to "recall" related past experiences. This approach significantly improves performance in data-sparse regions of the state space by providing necessary context for value estimation.

Introduction

Offline RL aims to learn optimal policies from fixed, pre-collected datasets without further environment interaction. While benchmarks like D4RL are standard, they often feature near-uniform state coverage. In contrast, real-world datasets are rarely balanced due to safety constraints or limited exploration. Critical or expert-level states are often rare, while trivial or safe states are over-represented, following Zipf’s law.

Current state-of-the-art methods, such as Conservative Q-Learning (CQL), fail in these scenarios for two primary reasons:

  1. State-Agnostic Pessimism: Uniform penalties across all state-action pairs create a "catch-22." Low pessimism leads to poor value estimation in rare states, while high pessimism prevents learning better-than-behavior policies in common states.
  2. Uniform Sampling: Standard training samples transitions uniformly, causing rare expert transitions to be "drowned out" by suboptimal, common transitions, leading to massive temporal difference (TD) errors in critical areas.

Methodology: RB-CQL

The authors propose RB-CQL, inspired by natural intelligence's ability to recall similar past events to handle novel situations. The method adds a retrieval layer to the standard CQL framework:

  • Auxiliary Dataset ($D_{aux}$): The agent accesses a large pool of experiences from various policies.
  • Retrieval Process: For a query state, the system uses Maximum Inner Product Search (MIPS) or Euclidean distance to find the top-$k$ most similar states in $D_{aux}$.
  • State Augmentation: The agent uses a concatenated state $s_{final}$, combining the original state with the average of the retrieved neighbor states.
  • Training: This augmented state provides context, allowing the agent to borrow information from "related" experiences to fill gaps in the imbalanced training set.
Visualization of the Four-room Navigation task and learned policies, illustrating how data imbalance affects state coverage and policy learning.

Experimental Evaluation

The method was evaluated on modified benchmarks to simulate real-world imbalance:

  • Four-room Navigation: A grid-world task used to visualize how pessimism levels affect performance in data-sparse rooms.
  • AntMaze (Medium & Large): D4RL tasks with skewed goal distributions (Easy to Hard+ levels).
  • MuJoCo Locomotion: Modified by mixing random and expert data in extreme ratios (up to 99% random) to create long-tail distributions.

Key Results and Insights

RB-CQL achieved superior performance in 10 out of 12 evaluated tasks. Key findings include:

  • Robustness to Imbalance: In AntMaze-medium, as imbalance increased from "Easy" to "Hard," standard CQL performance plummeted. RB-CQL remained stable, maintaining high performance even with extremely rare expert data.
  • The Failure of PER: Prioritized Experience Replay (PER), often used for rare transitions, actually worsened performance. The authors suggest that re-sampling introduces harmful distributional shifts in this context.
  • Trajectory Stitching: Qualitative analysis showed RB-CQL successfully "stitched" suboptimal early paths with expert-like behavior in final stages, even when expert data was nearly non-existent.
Quantitative results showing normalized returns across imbalance levels and the comparative performance of RB-CQL against baselines and PER.

Conclusion

RB-CQL addresses the challenges of imbalanced offline RL by leveraging non-parametric retrieval to smooth the state space. While it introduces computational overhead and depends on a diverse auxiliary dataset, it offers a robust solution for learning from real-world, long-tailed data distributions. Future work will explore scaling to high-dimensional inputs and more efficient auxiliary data construction.

Summary updated

BibTeX
@misc{jiang2023offline,
  author = {Li Jiang and Sijie Chen and Jielin Qiu and Haoran Xu and Wai Kin Chan and Zhao Ding},
  title = {{Offline Reinforcement Learning with Imbalanced Datasets}},
  year = {2023},
  eprint = {2307.02752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.02752},
}
Figure