Predictable MDP Abstraction for Unsupervised Model-Based RL

Seohong Park, Sergey Levine

Abstract

Predictable MDP Abstraction (PMA) is a novel unsupervised reinforcement learning framework designed to mitigate "model exploitation" in model-based RL. By restricting an agent's action space to only those transitions that are easy to model, PMA creates a simplified latent environment where dynamics are highly predictable. This approach enables robust, zero-shot task performance in challenging robotics environments, such as the MuJoCo Ant, where standard world models typically fail due to chaotic contact dynamics.

Introduction

Model-based reinforcement learning (MBRL) offers the promise of sample efficiency by allowing agents to "dream" or plan within a learned world model. However, a persistent challenge is model exploitation: when a dynamics model contains even minor inaccuracies, a planner will often find and exploit these errors, leading to behaviors that fail in the real world. This is particularly prevalent in complex, contact-rich environments where perfect modeling is nearly impossible. PMA addresses this by shifting the focus from modeling the entire environment to modeling only a predictable subset of it.

Methodology

The core of PMA is the transformation of a complex Markov Decision Process (MDP) into a "predictable" latent MDP. This is achieved through three jointly trained components:

  1. Action Decoder ($\pi_z$): Maps a latent action ($z$) to a real-world action ($a$), specifically filtering for actions that lead to predictable outcomes.
  2. Latent Predictive Model ($\hat{p}_z$): A dynamics model that predicts the next state based on the current state and the latent action.
  3. Exploration Policy ($\pi_e$): Encourages the agent to explore the environment while remaining within the predictable regime.
The Action Decoder maps the latent action space to a predictable subset of the original action space, excluding chaotic or unpredictable transitions.

The framework uses an information-theoretic objective to maximize mutual information between the next state and the latent action. This objective balances three goals: predictability (minimizing next-state uncertainty), diversity (ensuring distinct latent actions lead to different outcomes), and information gain (driving exploration toward uncertain but predictable states).

Experimental Evaluation

PMA was evaluated on seven MuJoCo robotics environments across 13 diverse tasks. After unsupervised pre-training, the agent's learned latent model was used with planners like MPPI to solve tasks zero-shot, without further real-world interaction.

Key Results and Insights

  • Solving the "Ant" Problem: The Ant environment is notoriously difficult for MBRL because the robot frequently flips over, creating chaotic dynamics. PMA learns to restrict the action space such that the Ant remains upright, resulting in a stable and reliable dynamics model.
  • Superior Zero-Shot Performance: PMA significantly outperformed strong baselines, including DADS and various exploration-based world models. It was the only unsupervised model-based method tested that successfully solved the Ant tasks.
  • Lossy Abstraction as a Feature: By intentionally "throwing away" chaotic parts of the action space, PMA achieves lower prediction error than standard models, even those with significantly more parameters.
Comparison of the Ant environment in the original MDP versus the Predictable MDP. PMA prevents the agent from entering chaotic states, such as flipping over.

Conclusion

Predictable MDP Abstraction demonstrates that "lossy" environment modeling can be a powerful tool for robust RL. By prioritizing predictability over total coverage, PMA avoids the pitfalls of model exploitation and enables effective zero-shot planning in complex robotics domains. Future work may explore extending this framework to high-dimensional pixel inputs and hybridizing it with model-free approaches to handle necessary but unpredictable maneuvers.

Summary updated

Abstract

A key component of model-based reinforcement learning (RL) is a dynamics model that predicts the outcomes of actions. Errors in this predictive model can degrade the performance of model-based controllers, and complex Markov decision processes (MDPs) can present exceptionally difficult prediction problems. To mitigate this issue, we propose predictable MDP abstraction (PMA): instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space that only permits predictable, easy-to-model actions, while covering the original state-action space as much as possible. As a result, model learning becomes easier and more accurate, which allows robust, stable model-based planning or model-based RL. This transformation is learned in an unsupervised manner, before any task is specified by the user. Downstream tasks can then be solved with model-based control in a zero-shot fashion, without additional environment interactions. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches in a range of benchmark environments. Our code and videos are available at https://seohong.me/projects/pma/
BibTeX
@inproceedings{park2023predictable,
  author = {Seohong Park and Sergey Levine},
  title = {{Predictable MDP Abstraction for Unsupervised Model-Based RL}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=pj3EXUk5Uj},
}
Figure