Abstract
Many control tasks exhibit similar dynamics that can be modeled as having common latent structure. Hidden-Parameter Markov Decision Processes (HiP-MDPs) explicitly model this structure to improve sample efficiency in multi-task settings. However, this setting makes strong assumptions on the observability of the state that limit its application in real-world scenarios with rich observation spaces. In this work, we leverage ideas of common structure from the HiP-MDP setting, and extend it to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings. Further, we provide transfer and generalization bounds based on task and state similarity, along with sample complexity bounds that depend on the aggregate number of samples across tasks, rather than the number of tasks, a significant improvement over prior work that use the same environment assumptions. To further demonstrate the efficacy of the proposed method, we empirically compare and show improvement over multi-task and meta-reinforcement learning baselines.
Introduction
In the quest to build versatile AI, Reinforcement Learning (RL) researchers face a daunting "double whammy": agents must handle high-dimensional visual inputs (like pixels from a camera) while simultaneously adapting to varying physical properties (like a robot moving from a carpeted floor to a slippery tile one). Most RL algorithms are "specialists" that excel at a single task in a static environment. However, real-world applications require "generalists" capable of bridging two major gaps:
- The Observability Gap: Real robots do not see their exact coordinates; they see noisy pixels. The Block MDP framework addresses this by assuming high-dimensional observations map to a simpler latent state.
- The Dynamics Gap: Physical parameters change. A Hidden-Parameter MDP (HiP-MDP) models this by assuming an unobserved variable (like gravity or friction) dictates the environment's rules.
Prior work typically addressed one or the other. This paper, presented at ICLR 2021, introduces the Hidden-Parameter Block MDP (HiP-BMDP), a framework for agents that can handle pixel-based observations in environments with changing, unobserved physical parameters.
The HiP-BMDP Framework
The core idea of HiP-BMDP is to learn a "Super-MDP"—a single, unified model where the only variance between tasks is a latent parameter, $\theta$. By encoding task-specific physical properties into this latent variable, the agent can learn representations that are robust to both visual clutter and changing dynamics.
Architecture and Methodology
The HiP-BMDP architecture consists of several key neural components designed to extract and utilize latent structure:
- State Encoder ($\phi$): Maps raw observations (pixels) into a latent state space $s$.
- Environment Encoder ($\psi$): Maps an environment ID to a hidden parameter $\theta$ representing the "physics" of that specific task.
- Universal Dynamics Model ($T$): A single model that predicts the next latent state $s'$ based on the current state, the action taken, and the task parameter $\theta$.
- Reward Model ($R$): Predicts rewards based on the latent state transitions.
A critical innovation is the use of a Bisimulation Metric Loss. Instead of predicting raw pixels, the agent learns to group states together if they lead to similar rewards and similar future states. By using the Wasserstein distance to measure task differences, the environment encoder is forced to map similar physics to similar values of $\theta$, ensuring a smooth and generalizable latent space.
Experimental Evaluation
The researchers evaluated HiP-BMDP using the DeepMind Control Suite (DMC). They created "families" of tasks by varying specific physical parameters, such as the mass of a pole in Cartpole-Swingup or the torso size in Cheetah-Run. The agents were tested in two regimes:
- Interpolation: Tasks with parameters within the training range.
- Extrapolation: Tasks with parameters outside the training range (e.g., a much heavier pole than previously seen).
Key Results
- Zero-Shot Generalization: In multi-task settings, HiP-BMDP significantly outperformed baselines like DeepMDP and PCGrad on extrapolation tasks, maintaining high rewards even under extreme physical parameters.
- Faster Adaptation: In Meta-RL experiments, adding the HiP-BMDP loss allowed agents to reach threshold rewards much faster than standard Meta-RL algorithms like PEARL.
- Theoretical Efficiency: The authors proved that performance depends on the aggregate number of samples across all tasks, rather than the number of unique tasks, suggesting that learning shared structure is fundamentally more efficient.
Conclusion
HiP-BMDP provides a robust bridge between pixel-level perception and varying physical dynamics. By treating the task parameter as a "knob" within a universal dynamics model, the framework allows for superior generalization and sample efficiency. Future work may focus on inferring the hidden $\theta$ parameter directly from interaction frames, further reducing the reliance on explicit environment IDs during training.
Summary updated