Abstract
Many control tasks exhibit similar dynamics that can be modeled as having common latent structure. Hidden-Parameter Markov Decision Processes (HiP-MDPs) explicitly model this structure to improve sample efficiency in multi-task settings. However, this setting makes strong assumptions on the observability of the state that limit its application in real-world scenarios with rich observation spaces. In this work, we leverage ideas of common structure from the HiP-MDP setting, and extend it to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings. Further, we provide transfer and generalization bounds based on task and state similarity, along with sample complexity bounds that depend on the aggregate number of samples across tasks, rather than the number of tasks, a significant improvement over prior work that use the same environment assumptions. To further demonstrate the efficacy of the proposed method, we empirically compare and show improvement over multi-task and meta-reinforcement learning baselines.
Introduction
In Multi-Task Reinforcement Learning (MTRL), an agent is tasked with performing well across a variety of related environments. This presents two primary challenges: high-dimensional observations, such as pixels, where much of the information is irrelevant to the task; and varying dynamics, where tasks share goals but differ in physical properties (e.g., varying friction or mass).
Existing approaches often treat tasks as independent or assume full observability of the true state. This work introduces the Hidden-Parameter Block MDP (HiP-BMDP), bridging the gap between Hidden-Parameter MDPs—which model shared latent structure across tasks—and Block MDPs—which model rich observations generated from latent states.
Methods
The HiP-BMDP framework learns robust state abstractions by grouping observations that are "behaviorally equivalent" using bisimulation metrics. Two states are considered bisimilar if they yield the same immediate reward and lead to the same distribution of future states for any given action.
Architecture and Latent Task Parameters
The model employs an environment encoder to map task IDs to a latent parameter $\theta$. This parameter conditions a universal dynamics model, allowing it to adapt to specific environment physics. The training objective combines a standard model-learning error with a novel task bisimulation loss, ensuring that the distance between learned task parameters reflects the differences in their transition dynamics.
Theoretical Contributions
A key theoretical result of this work is that sample complexity depends on the aggregate number of samples across all tasks, rather than the number of tasks themselves. This provides a formal justification for the improved sample efficiency observed in multi-task settings.
Experimental Evaluation
The framework was evaluated on the DeepMind Control Suite (DMC), using tasks with varying physical parameters such as mass, torso size, and friction.
- Zero-Shot Generalization: HiP-BMDP significantly outperformed baselines like DeepMDP and PCGrad on unseen "extrapolation" tasks, maintaining high performance even as environment parameters deviated significantly from training data.
- Meta-RL Adaptation: When integrated with the PEARL algorithm, the approach demonstrated faster convergence to target rewards in Walker and Cartpole environments.
- Robustness: The model remained effective even under relaxed Block MDP assumptions, such as partially observable "sticky observations."
Conclusion
The HiP-BMDP framework provides a robust method for multi-task and meta-reinforcement learning by leveraging latent task structure and bisimulation-based state abstractions. While current implementations require environment IDs during training, future work will explore automated environment identification and tighter theoretical bounds.
Summary updated