Model-based Offline Reinforcement Learning with Count-based Conservatism

Byeongchan Kim, Min-hwan Oh

TL;DR

Count-MORL is a novel model-based offline reinforcement learning (RL) framework that utilizes state-action counts to enforce "pessimism" during training. By penalizing rewards for rare or unseen actions using an innovative hash-based counting mechanism, it effectively bridges the gap between theoretical RL guarantees and practical performance. The method significantly outperforms existing state-of-the-art algorithms on the D4RL benchmark.

The Challenge: Distributional Shift in Offline RL

Offline RL is often considered the "holy grail" for real-world applications like autonomous driving or healthcare. It allows an agent to learn from historical data without requiring trial-and-error interactions that could be dangerous or costly in the real world. However, offline RL suffers from a massive hurdle: distributional shift.

When an agent evaluates a policy, it might consider actions that were not present in the original dataset (Out-of-Distribution, or OOD). Because the model hasn't seen these actions, it often makes over-optimistic predictions about their value, leading the agent to "hallucinate" high-reward paths that don't actually exist.

To fix this, researchers rely on conservatism (or pessimism). Essentially, the agent is programmed to assume that if it hasn't seen a state-action pair many times, it should be treated as a bad choice. While many existing methods use complex uncertainty estimates—like the variance of a neural network ensemble—these can be computationally heavy and difficult to map back to the clean "count-based" proofs found in foundational RL theory.

Introducing Count-MORL

The authors introduce Count-MORL (Count-based Model-based Offline RL). Its core innovation lies in using the frequency of state-action pairs in the offline dataset to directly quantify model error and enforce conservatism.

1. The Count-Based Penalty

The researchers mathematically prove that model estimation error is inversely proportional to the square root of the state-action count ($1/\sqrt{n}$). They use this relationship to create a "penalized reward" function: the more frequent a state-action pair is in the dataset, the lower the penalty applied to its reward.

2. Hash-Code Counting in Continuous Spaces

In continuous environments (like a robot's joint angles), you cannot simply count occurrences because every state is technically unique. To solve this, Count-MORL employs an autoencoder.

Architecture of the Count-MORL framework, illustrating the dynamics model and the autoencoder used for hash-based counting.

The system takes the activations from a bottleneck layer in the autoencoder, binarizes them into a $d$-dimensional hash code, and counts how often each unique hash appears. This allows the algorithm to group similar continuous states into discrete, countable bins.

3. Managing Uncertainty with Ensembles

Because Count-MORL uses an ensemble of models, it generates multiple count estimates. The authors propose three distinct ways to aggregate these counts depending on the dataset:

Algorithm for computing the approximate count using Lower Confidence (LC), Average (AVG), or Upper Confidence (UC) estimates.
  • LC (Lower Confidence): High pessimism. Best for low-quality or random data where caution is paramount.
  • AVG (Average): A balanced, middle-ground approach.
  • UC (Upper Confidence): Low pessimism. Best for high-quality (expert) data where you want the agent to stay close to the demonstrated behavior without being overly penalized.

Experimental Results

The researchers evaluated Count-MORL on two main fronts: a Grid-World environment (as a sanity check to prove hash-based counting accurately reflects true frequencies) and the industry-standard D4RL Benchmark (featuring MuJoCo environments like HalfCheetah, Hopper, and Walker2d across various data qualities).

Count-MORL achieved impressive results across the board:

  • State-of-the-Art Performance: It outperformed existing leaders like MOPO, COMBO, and CQL on the D4RL MuJoCo-v2 datasets.
  • Robustness: It achieved the best or highly competitive performance in 10 out of 12 tested settings.
  • High Average Scores: On the MuJoCo-v2 average, Count-MORL scored 78.8, significantly higher than MOReL (64.4) and CQL (62.0).
  • Sample Efficiency: The agent learned optimal policies with fewer training episodes compared to previous model-based methods.

Key Insights

  • Theory Meets Practice: Count-based methods are a staple of online RL to encourage exploration. This paper flips the script, demonstrating they are equally powerful for offline RL to enforce conservatism.
  • Data Quality Dictates Pessimism: The type of count estimation matters. If your offline data is "trash" (random), you should be extra pessimistic (LC count). If your data is "gold" (expert), you should be less pessimistic (UC count) to avoid straying from the expert's path.
  • Simplicity Scales: Despite the complexity of deep RL, a simple count-based penalty derived from a hash code was enough to beat much more mathematically "heavy" uncertainty quantification methods.

Limitations and Future Work

While powerful, Count-MORL has a few areas for potential improvement. The performance is somewhat sensitive to the dimension of the hash code ($d$) and the quality of the autoencoder's latent space. Additionally, like most model-based RL, it requires tuning hyperparameters such as the rollout length ($H$) and the penalty coefficient ($\beta$). Finally, the current theory assumes the reward function is known, though the authors note it can be learned alongside the transition model in practice.

Future work could involve automating the choice between LC, AVG, and UC counts based on an automated assessment of the dataset's quality, further streamlining the deployment of offline RL in real-world scenarios.

Summary updated

Abstract

In this paper, we present a model-based offline reinforcement learning method that integrates count-based conservatism, named $\texttt{Count-MORL}$. Our method utilizes the count estimates of state-action pairs to quantify model estimation error, marking the first algorithm of demonstrating the efficacy of count-based conservatism in model-based offline deep RL to the best of our knowledge. For our proposed method, we first show that the estimation error is inversely proportional to the frequency of state-action pairs. Secondly, we demonstrate that the learned policy under the count-based conservative model offers near-optimality performance guarantees. Through extensive numerical experiments, we validate that $\texttt{Count-MORL}$ with hash code implementation significantly outperforms existing offline RL algorithms on the D4RL benchmark datasets. The code is accessible at https://github.com/oh-lab/Count-MORL.
BibTeX
@inproceedings{kim2023model,
  author = {Byeongchan Kim and Min-hwan Oh},
  title = {{Model-based Offline Reinforcement Learning with Count-based Conservatism}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=T5VlejGx7f},
}
Figure