Abstract
Predictable MDP Abstraction (PMA) is a novel unsupervised reinforcement learning framework designed to mitigate "model exploitation" in model-based RL. By restricting an agent's action space to only those transitions that are easy to model, PMA creates a simplified latent environment where dynamics are highly predictable. This approach enables robust, zero-shot task performance in challenging robotics environments, such as the MuJoCo Ant, where standard world models typically fail due to chaotic contact dynamics.
Introduction
Model-based reinforcement learning (MBRL) offers the promise of sample efficiency by allowing agents to "dream" or plan within a learned world model. However, a persistent challenge is model exploitation: when a dynamics model contains even minor inaccuracies, a planner will often find and exploit these errors, leading to behaviors that fail in the real world. This is particularly prevalent in complex, contact-rich environments where perfect modeling is nearly impossible. PMA addresses this by shifting the focus from modeling the entire environment to modeling only a predictable subset of it.
Methodology
The core of PMA is the transformation of a complex Markov Decision Process (MDP) into a "predictable" latent MDP. This is achieved through three jointly trained components:
- Action Decoder ($\pi_z$): Maps a latent action ($z$) to a real-world action ($a$), specifically filtering for actions that lead to predictable outcomes.
- Latent Predictive Model ($\hat{p}_z$): A dynamics model that predicts the next state based on the current state and the latent action.
- Exploration Policy ($\pi_e$): Encourages the agent to explore the environment while remaining within the predictable regime.
The framework uses an information-theoretic objective to maximize mutual information between the next state and the latent action. This objective balances three goals: predictability (minimizing next-state uncertainty), diversity (ensuring distinct latent actions lead to different outcomes), and information gain (driving exploration toward uncertain but predictable states).
Experimental Evaluation
PMA was evaluated on seven MuJoCo robotics environments across 13 diverse tasks. After unsupervised pre-training, the agent's learned latent model was used with planners like MPPI to solve tasks zero-shot, without further real-world interaction.
Key Results and Insights
- Solving the "Ant" Problem: The Ant environment is notoriously difficult for MBRL because the robot frequently flips over, creating chaotic dynamics. PMA learns to restrict the action space such that the Ant remains upright, resulting in a stable and reliable dynamics model.
- Superior Zero-Shot Performance: PMA significantly outperformed strong baselines, including DADS and various exploration-based world models. It was the only unsupervised model-based method tested that successfully solved the Ant tasks.
- Lossy Abstraction as a Feature: By intentionally "throwing away" chaotic parts of the action space, PMA achieves lower prediction error than standard models, even those with significantly more parameters.
Conclusion
Predictable MDP Abstraction demonstrates that "lossy" environment modeling can be a powerful tool for robust RL. By prioritizing predictability over total coverage, PMA avoids the pitfalls of model exploitation and enables effective zero-shot planning in complex robotics domains. Future work may explore extending this framework to high-dimensional pixel inputs and hybridizing it with model-free approaches to handle necessary but unpredictable maneuvers.
Summary updated