Abstract
Standard offline reinforcement learning (RL) algorithms often struggle with real-world datasets where state coverage is heavily imbalanced, typically following a power-law distribution. This paper introduces Retrieval-Based Conservative Q-Learning (RB-CQL), which augments agents with a non-parametric retrieval process to "recall" related past experiences. This approach significantly improves performance in data-sparse regions of the state space by providing necessary context for value estimation.
Introduction
Offline RL aims to learn optimal policies from fixed, pre-collected datasets without further environment interaction. While benchmarks like D4RL are standard, they often feature near-uniform state coverage. In contrast, real-world datasets are rarely balanced due to safety constraints or limited exploration. Critical or expert-level states are often rare, while trivial or safe states are over-represented, following Zipf’s law.
Current state-of-the-art methods, such as Conservative Q-Learning (CQL), fail in these scenarios for two primary reasons:
- State-Agnostic Pessimism: Uniform penalties across all state-action pairs create a "catch-22." Low pessimism leads to poor value estimation in rare states, while high pessimism prevents learning better-than-behavior policies in common states.
- Uniform Sampling: Standard training samples transitions uniformly, causing rare expert transitions to be "drowned out" by suboptimal, common transitions, leading to massive temporal difference (TD) errors in critical areas.
Methodology: RB-CQL
The authors propose RB-CQL, inspired by natural intelligence's ability to recall similar past events to handle novel situations. The method adds a retrieval layer to the standard CQL framework:
- Auxiliary Dataset ($D_{aux}$): The agent accesses a large pool of experiences from various policies.
- Retrieval Process: For a query state, the system uses Maximum Inner Product Search (MIPS) or Euclidean distance to find the top-$k$ most similar states in $D_{aux}$.
- State Augmentation: The agent uses a concatenated state $s_{final}$, combining the original state with the average of the retrieved neighbor states.
- Training: This augmented state provides context, allowing the agent to borrow information from "related" experiences to fill gaps in the imbalanced training set.
Experimental Evaluation
The method was evaluated on modified benchmarks to simulate real-world imbalance:
- Four-room Navigation: A grid-world task used to visualize how pessimism levels affect performance in data-sparse rooms.
- AntMaze (Medium & Large): D4RL tasks with skewed goal distributions (Easy to Hard+ levels).
- MuJoCo Locomotion: Modified by mixing random and expert data in extreme ratios (up to 99% random) to create long-tail distributions.
Key Results and Insights
RB-CQL achieved superior performance in 10 out of 12 evaluated tasks. Key findings include:
- Robustness to Imbalance: In AntMaze-medium, as imbalance increased from "Easy" to "Hard," standard CQL performance plummeted. RB-CQL remained stable, maintaining high performance even with extremely rare expert data.
- The Failure of PER: Prioritized Experience Replay (PER), often used for rare transitions, actually worsened performance. The authors suggest that re-sampling introduces harmful distributional shifts in this context.
- Trajectory Stitching: Qualitative analysis showed RB-CQL successfully "stitched" suboptimal early paths with expert-like behavior in final stages, even when expert data was nearly non-existent.
Conclusion
RB-CQL addresses the challenges of imbalanced offline RL by leveraging non-parametric retrieval to smooth the state space. While it introduces computational overhead and depends on a diverse auxiliary dataset, it offers a robust solution for learning from real-world, long-tailed data distributions. Future work will explore scaling to high-dimensional inputs and more efficient auxiliary data construction.
Summary updated