Model-Based Offline Policy Optimization with Adversarial Network

Junming Yang, Xingguo Chen, Shengyuan Wang, Bolei Zhang

Abstract

Offline Reinforcement Learning (RL) offers a pathway to leverage static datasets for policy optimization, yet it is frequently hampered by distributional shift. When agents encounter state-action pairs outside the training distribution, transition models often produce erroneous, over-optimistic predictions. This paper introduces MOAN (Model-based Offline policy optimization with Adversarial Network), a framework that treats transition modeling as a two-player adversarial game. By training a generator to produce transitions and a discriminator to identify real data, MOAN establishes a robust, theoretically grounded uncertainty measure. Experimental results on the D4RL benchmark demonstrate that MOAN achieves state-of-the-art performance, offering superior generalization and more reliable reward penalization than existing heuristic-based methods.

1. Introduction

Reinforcement Learning typically necessitates millions of environmental interactions, a requirement that is often impractical or hazardous in real-world applications such as autonomous driving or clinical healthcare. Offline RL addresses this by learning exclusively from a fixed "logging dataset" collected by a prior policy.

However, the central challenge in Offline RL is the distributional shift. As the learning agent attempts to improve upon the behavior policy, it inevitably queries the transition model in regions of the state-action space not covered by the dataset. In these "unseen" regions, standard transition models often fail, leading the agent to pursue "fake" high rewards. While existing solutions attempt to be conservative by penalizing rewards based on heuristic uncertainty estimates, these penalties are often either unreliable or excessively cautious, stifling the agent's ability to learn effectively.

2. Methodology

MOAN proposes a novel approach by framing transition modeling as a generative adversarial process. The framework consists of two primary components: Adversarial Transition Modeling and Reward Reshaping.

2.1 Adversarial Transition Modeling

Instead of relying solely on maximum likelihood estimation (MLE), MOAN utilizes a Generator—comprising an ensemble of Gaussian neural networks—to predict subsequent states and rewards. Simultaneously, a Discriminator is trained to distinguish between authentic transitions from the logging dataset and synthetic transitions produced by the generator.

This adversarial setup forces the generator to produce diverse samples that remain indistinguishable from real-world data. By competing against the discriminator, the generator learns a more robust representation of the environment's physics, significantly improving its ability to generalize to underrepresented regions of the state space.

2.2 Reward Reshaping with Discrepancy Measures

To ensure policy safety and prevent over-optimism, MOAN incorporates a sophisticated reward penalization scheme. Unlike previous methods that rely exclusively on model variance (internal uncertainty), MOAN leverages the Discriminator's output as a direct measure of "out-of-distribution" (OOD) discrepancy (external uncertainty).

The final reshaped reward is calculated by subtracting penalties for both internal model variance and the discriminator's discrepancy score from the predicted reward. This dual-penalty approach provides a tighter lower bound on the policy's performance in the actual environment. The resulting policy is then optimized using Soft Actor-Critic (SAC) on a hybrid dataset of real and adversarially-vetted synthetic samples.

3. Experimental Evaluation

3.1 Datasets and Setup

The researchers evaluated MOAN using the D4RL benchmark, the standard for offline RL research. Testing spanned three MuJoCo environments: HalfCheetah, Hopper, and Walker2d. To ensure a comprehensive evaluation, the model was tested across four data quality levels: random, medium, medium-replay, and medium-expert.

3.2 Key Results

  • State-of-the-Art Performance: MOAN achieved the highest scores in 7 out of the 12 D4RL task configurations, outperforming established baselines such as MOPO, CQL, and COMBO.
  • Enhanced Generalization: Through t-SNE visualizations, the authors demonstrated that MOAN’s generated states are significantly more consistent with the real-world distribution than those of its predecessors. While models like MOPO often drifted into unrealistic territory, MOAN remained anchored to the underlying data distribution.
  • Learning Efficiency: The algorithm exhibits rapid convergence, frequently reaching peak performance within approximately 20,000 training steps.
  • Robustness in Low-Dimensional Spaces: MOAN showed exceptional performance in the "Hopper" tasks. The authors suggest that while other models become overly conservative in lower-dimensional state spaces, MOAN’s adversarial approach allows for more effective and safe exploration.

4. Discussion and Limitations

MOAN's primary strength lies in its use of the discriminator as a "coach" rather than just a training tool. By providing a learned measure of "realness," the discriminator offers a more nuanced uncertainty quantification than simple statistical heuristics. Furthermore, the framework prioritizes model generalization over pure conservatism, leading to more capable agents.

However, the approach is not without trade-offs. The primary limitation is computational complexity; the addition of a discriminator and the adversarial training loop increases the overhead compared to standard supervised models. Additionally, while MOAN dominated most categories, it was slightly outperformed by certain model-free methods on specific "medium-expert" datasets, suggesting that the transition model may still struggle with highly specialized expert trajectories.

5. Conclusion

MOAN introduces a theoretically grounded and highly effective adversarial framework for offline reinforcement learning. By integrating a discriminator into the transition modeling process, it successfully mitigates the risks of distributional shift while maintaining the flexibility needed for policy improvement. Future work will focus on integrating model-free techniques to better handle expert data and exploring more complex discriminator architectures to further refine uncertainty quantification.

Summary updated

BibTeX
@inproceedings{yang2023model,
  author = {Junming Yang and Xingguo Chen and Shengyuan Wang and Bolei Zhang},
  title = {{Model-Based Offline Policy Optimization with Adversarial Network}},
  booktitle = {ECAI},
  year = {2023},
  doi = {10.3233/faia230597},
}
Figure