Introduction
This paper introduces RepB-SDE, a novel framework for offline model-based reinforcement learning that tackles the "distribution shift" problem by learning a balanced state representation. By aligning the data distribution with the target policy's discounted stationary distribution using the "DualDICE trick," the authors overcome the "curse of horizon" that plagued previous balancing methods, achieving state-of-the-art performance on standard offline RL benchmarks.
The Challenge of Distribution Shift
Offline Reinforcement Learning (RL) is highly desirable for real-world applications like robotics or healthcare, where letting an agent learn through trial-and-error (online) is too dangerous or expensive. Instead, the goal is to learn an optimal policy from a fixed, pre-collected dataset.
The central challenge in this setting is distribution shift: the dataset was collected by a "behavior policy," but we want to optimize a different "target policy." If our environment model is only accurate on the behavior data, it might make wild, over-optimistic predictions in regions the target policy visits. This leads the agent to "hallucinate" high rewards and fail miserably in practice.
Previous attempts to fix this, such as Representation Balancing MDPs (RepBM), tried to make the model's internal representations invariant across policies at every timestep. However, this approach suffers from the curse of horizon: errors in importance sampling compound exponentially over time, making the method ineffective for long-duration tasks.
Representation Balancing with RepB-SDE
To address these limitations, the authors propose Representation Balancing with Stationary Distribution Estimation (RepB-SDE). The core idea is to regularize the model's representation learning so that the distribution of the training data looks similar to the distribution of states the target policy is expected to visit in the long run.
Key technical components of this approach include:
- Stationary Distribution Alignment: Instead of balancing at every step, RepB-SDE balances the discounted stationary distribution. This looks at the "average" occupancy of states, which is much more stable over long horizons.
- The DualDICE Trick: Since we lack samples from the target policy's distribution, the authors use a change-of-variables technique from recent "DICE" (Distribution Correction Estimation) literature. This allows them to estimate the distance between distributions using only the offline dataset.
- RKHS and IPM: They measure the distance between distributions using an Integral Probability Metric (IPM) in a Reproducing Kernel Hilbert Space (RKHS). This provides a tractable, closed-form objective function that can be plugged directly into a neural network's loss function.
- Model-Based Policy Optimization: This robust model is integrated into a standard policy optimization loop (similar to MBPO), using a bootstrap ensemble of models and pessimistic reward penalties to handle uncertainty.
Experimental Results
The authors evaluated their method on two primary fronts: Off-Policy Evaluation (OPE) and standard Offline RL Benchmarks.
Off-Policy Evaluation (OPE) Accuracy
The framework was tested on CartPole-v0, Acrobot-v1, and a complex HIV simulator to see how accurately the model could predict the return of a target policy.
As shown above, RepB-SDE significantly reduced the Mean Squared Error (MSE) of return estimates compared to the original RepBM and other baselines. This improvement becomes especially pronounced as the "off-policyness" (the gap between behavior and target policies) increases.
D4RL Benchmark Performance
For policy optimization, the method was evaluated on the D4RL MuJoCo suite (Walker2d, Hopper, and HalfCheetah) across various data quality levels: Random, Medium, Medium-Replay, and Medium-Expert.
RepB-SDE achieved state-of-the-art or highly competitive results across these tasks. Notably, in Walker2d-Medium, it scored 72.1, vastly outperforming the naive model-based baseline (5.5) and the previous SOTA MOPO (-0.1). It also showed consistent improvements over "RP" (a baseline with reward penalties but no representation balancing), proving that the balancing objective is the key driver of performance.
Key Insights
- Robustness via Representation: Most offline RL research focuses on making the policy conservative. This paper argues that making the model robust through representation balancing provides a much stronger foundation for any subsequent policy optimization.
- Solving the Curse of Horizon: By shifting from step-wise importance sampling to stationary distribution estimation, the authors transformed a mathematically "hard" problem (exponential variance) into a manageable one.
- Trade-off between Fitness and Invariance: The objective function explicitly balances two terms: how well the model fits the observed data (likelihood) and how invariant the representation is across distributions (IPM). This prevents the model from becoming "too simple" just to satisfy the balancing requirement.
Limitations and Future Work
- Hopper-v2 Performance: The method was less dominant on the Hopper environment. The authors suggest this might be due to the specific dynamics of Hopper requiring even more aggressive conservative training techniques (like behavior regularization) that weren't the focus of this paper.
- Representation Reuse: Currently, the balanced representation is used to train a robust model, but the policy itself doesn't explicitly leverage this representation. Incorporating the learned features directly into the policy could be a promising next step.
- Computational Complexity: Calculating the IPM in an RKHS can be computationally intensive for very large datasets, suggesting a need for more efficient kernel approximations in the future.
Summary updated