Representation Balancing Offline Model-based Reinforcement Learning

Byung-Jun Lee, Jongmin Lee, Kee-Eung KimICLR 2021 · OpenReview

Introduction

This paper introduces RepB-SDE, a novel framework for offline model-based reinforcement learning that tackles the "distribution shift" problem by learning a balanced state representation. By aligning the data distribution with the target policy's discounted stationary distribution using the "DualDICE trick," the authors overcome the "curse of horizon" that plagued previous balancing methods, achieving state-of-the-art performance on standard offline RL benchmarks.

The Challenge of Distribution Shift

Offline Reinforcement Learning (RL) is highly desirable for real-world applications like robotics or healthcare, where letting an agent learn through trial-and-error (online) is too dangerous or expensive. Instead, the goal is to learn an optimal policy from a fixed, pre-collected dataset.

The central challenge in this setting is distribution shift: the dataset was collected by a "behavior policy," but we want to optimize a different "target policy." If our environment model is only accurate on the behavior data, it might make wild, over-optimistic predictions in regions the target policy visits. This leads the agent to "hallucinate" high rewards and fail miserably in practice.

Previous attempts to fix this, such as Representation Balancing MDPs (RepBM), tried to make the model's internal representations invariant across policies at every timestep. However, this approach suffers from the curse of horizon: errors in importance sampling compound exponentially over time, making the method ineffective for long-duration tasks.

Representation Balancing with RepB-SDE

To address these limitations, the authors propose Representation Balancing with Stationary Distribution Estimation (RepB-SDE). The core idea is to regularize the model's representation learning so that the distribution of the training data looks similar to the distribution of states the target policy is expected to visit in the long run.

Key technical components of this approach include:

  • Stationary Distribution Alignment: Instead of balancing at every step, RepB-SDE balances the discounted stationary distribution. This looks at the "average" occupancy of states, which is much more stable over long horizons.
  • The DualDICE Trick: Since we lack samples from the target policy's distribution, the authors use a change-of-variables technique from recent "DICE" (Distribution Correction Estimation) literature. This allows them to estimate the distance between distributions using only the offline dataset.
  • RKHS and IPM: They measure the distance between distributions using an Integral Probability Metric (IPM) in a Reproducing Kernel Hilbert Space (RKHS). This provides a tractable, closed-form objective function that can be plugged directly into a neural network's loss function.
  • Model-Based Policy Optimization: This robust model is integrated into a standard policy optimization loop (similar to MBPO), using a bootstrap ensemble of models and pessimistic reward penalties to handle uncertainty.

Experimental Results

The authors evaluated their method on two primary fronts: Off-Policy Evaluation (OPE) and standard Offline RL Benchmarks.

Off-Policy Evaluation (OPE) Accuracy

The framework was tested on CartPole-v0, Acrobot-v1, and a complex HIV simulator to see how accurately the model could predict the return of a target policy.

OPE Accuracy across environments

As shown above, RepB-SDE significantly reduced the Mean Squared Error (MSE) of return estimates compared to the original RepBM and other baselines. This improvement becomes especially pronounced as the "off-policyness" (the gap between behavior and target policies) increases.

D4RL Benchmark Performance

For policy optimization, the method was evaluated on the D4RL MuJoCo suite (Walker2d, Hopper, and HalfCheetah) across various data quality levels: Random, Medium, Medium-Replay, and Medium-Expert.

D4RL MuJoCo Benchmark Results

RepB-SDE achieved state-of-the-art or highly competitive results across these tasks. Notably, in Walker2d-Medium, it scored 72.1, vastly outperforming the naive model-based baseline (5.5) and the previous SOTA MOPO (-0.1). It also showed consistent improvements over "RP" (a baseline with reward penalties but no representation balancing), proving that the balancing objective is the key driver of performance.

Key Insights

  • Robustness via Representation: Most offline RL research focuses on making the policy conservative. This paper argues that making the model robust through representation balancing provides a much stronger foundation for any subsequent policy optimization.
  • Solving the Curse of Horizon: By shifting from step-wise importance sampling to stationary distribution estimation, the authors transformed a mathematically "hard" problem (exponential variance) into a manageable one.
  • Trade-off between Fitness and Invariance: The objective function explicitly balances two terms: how well the model fits the observed data (likelihood) and how invariant the representation is across distributions (IPM). This prevents the model from becoming "too simple" just to satisfy the balancing requirement.

Limitations and Future Work

  • Hopper-v2 Performance: The method was less dominant on the Hopper environment. The authors suggest this might be due to the specific dynamics of Hopper requiring even more aggressive conservative training techniques (like behavior regularization) that weren't the focus of this paper.
  • Representation Reuse: Currently, the balanced representation is used to train a robust model, but the policy itself doesn't explicitly leverage this representation. Incorporating the learned features directly into the policy could be a promising next step.
  • Computational Complexity: Calculating the IPM in an RKHS can be computationally intensive for very large datasets, suggesting a need for more efficient kernel approximations in the future.

Summary updated

Abstract

One of the main challenges in offline and off-policy reinforcement learning is to cope with the distribution shift that arises from the mismatch between the target policy and the data collection policy. In this paper, we focus on a model-based approach, particularly on learning the representation for a robust model of the environment under the distribution shift, which has been first studied by Representation Balancing MDP (RepBM). Although this prior work has shown promising results, there are a number of shortcomings that still hinder its applicability to practical tasks. In particular, we address the curse of horizon exhibited by RepBM, rejecting most of the pre-collected data in long-term tasks. We present a new objective for model learning motivated by recent advances in the estimation of stationary distribution corrections. This effectively overcomes the aforementioned limitation of RepBM, as well as naturally extending to continuous action spaces and stochastic policies. We also present an offline model-based policy optimization using this new objective, yielding the state-of-the-art performance in a representative set of benchmark offline RL tasks.
BibTeX
@inproceedings{lee2021representation,
  author = {Byung-Jun Lee and Jongmin Lee and Kee-Eung Kim},
  title = {{Representation Balancing Offline Model-based Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=QpNz8r_Ri2Y},
}
Figure