Learning Robust State Abstractions for Hidden-Parameter Block MDPs

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, Joelle Pineau

Overview

In the quest for versatile artificial intelligence, reinforcement learning (RL) agents must learn behaviors that generalize across different environments. However, many control tasks exhibit similar dynamics that can be modeled as having a common latent structure. While existing frameworks like Hidden-Parameter Markov Decision Processes (HiP-MDPs) explicitly model this structure to improve sample efficiency in multi-task settings, they often make strong assumptions about state observability. This limits their application in real-world scenarios with rich, high-dimensional observation spaces, such as pixel-based camera feeds.

This paper introduces the Hidden-Parameter Block MDP (HiP-BMDP), a novel framework that enables RL agents to generalize across tasks with varying physics and high-dimensional visual inputs. By learning a task-aware state abstraction based on bisimulation metrics, the authors demonstrate superior zero-shot transfer and faster adaptation compared to state-of-the-art multi-task and meta-RL baselines.

The Challenge: High-Dimensionality and Varying Dynamics

Real-world robotic control faces two major hurdles when attempting to generalize learned behaviors:

  1. High-Dimensional Observations: Robots often perceive the world through cameras. These pixel-based observations contain a massive amount of "noise" that is irrelevant to the actual task, making it difficult to isolate the true state of the environment.
  2. Varying Dynamics: A robot might need to execute a running gait on grass, sand, or ice. While the overarching "task" remains the same, the underlying physics—such as friction, gravity, or mass—change significantly. These varying physical properties act as "hidden parameters."
Contrasting discrete Super-MDPs with the continuous, parameterized HiP-MDP framework.

Existing Multi-Task Reinforcement Learning (MTRL) often assumes the agent can see the true state directly or treats tasks as discrete, isolated entities (a Super-MDP). This paper bridges the gap by combining Hidden-Parameter MDPs (which model changing physics as a continuum) with Block MDPs (which model latent states hidden behind rich observations). The goal is to learn a representation robust enough to handle new, unseen physical variations without needing to retrain from scratch.

Methodology: The HiP-BMDP Framework

The authors propose the HiP-BMDP framework alongside a gradient-based representation learning algorithm designed to disentangle task-specific dynamics from complex observations.

The HiP-BMDP architecture featuring state and environment encoders alongside a universal dynamics model.

The core architecture consists of three primary components:

  • State Encoder ($\phi$): Maps high-dimensional observations (e.g., stacked pixel frames) to a compact, latent state representation.
  • Environment Encoder ($\psi$): Maps an environment ID to a task-specific hidden parameter embedding, $\theta$.
  • Universal Dynamics Model ($T$): Predicts the next latent state given the current latent state, the chosen action, and the task parameter $\theta$.

The Key Innovation: Task Bisimulation Loss

To ensure the learned representations are meaningful, the researchers utilize bisimulation metrics to group states that behave similarly. Crucially, they introduce a novel loss function that forces the distance between two task embeddings ($\theta_1$ and $\theta_2$) to match the maximum difference in their transition dynamics, measured via the Wasserstein distance.

This ensures the latent task space is "smooth." If two tasks have similar underlying physics, their embeddings will be close together in the latent space. This smoothness is what allows the agent to effectively interpolate or extrapolate its behavior to new, unseen physical settings.

Experimental Setup and Key Insights

The method was rigorously evaluated using the DeepMind Control Suite (DMC), a standard benchmark for continuous control. The authors created "families" of MDPs by varying specific physical parameters across different environments (e.g., varying pole mass in Cartpole-Swingup, torso size in Cheetah-Run, friction in Walker-Run, and finger length in Finger-Spin).

The models were tested in two distinct regimes:

  • Interpolation: Evaluating the agent on parameter values within the range seen during training.
  • Extrapolation: Evaluating the agent on parameter values outside the training range (e.g., controlling a much heavier pole than ever seen before).

Key Results

  • Zero-Shot Generalization: In the MTRL setting, HiP-BMDP significantly outperformed baselines like DeepMDP, PCGrad, and GradNorm on extrapolation tasks. It maintained higher episodic rewards even when environment parameters were pushed far beyond the training distribution.
  • Meta-RL Speedup: When integrated into PEARL (a popular meta-RL algorithm), the HiP-BMDP objective allowed the agent to adapt to new environments much faster, reaching threshold rewards in significantly fewer steps.
  • Aggregate Sample Complexity: Theoretically, the authors prove that the sample complexity of their method depends on the total number of samples across all tasks, rather than the number of individual tasks. This is a major theoretical improvement, suggesting that exposing the agent to more tasks actually accelerates learning by revealing the underlying shared structure.
  • Robustness to Partial Observability: The model remains effective even when the strict "Block MDP" assumption is relaxed (e.g., when the agent occasionally sees a frame from the past), demonstrating robustness to "sticky" observations.

Limitations and Future Work

While HiP-BMDP offers a powerful approach to robust state abstraction, the authors note a few limitations. Currently, the implementation relies on knowing when the environment has changed via an explicit environment ID. Future work could incorporate an automated "environment identification" procedure to cluster data without these labels.

Additionally, the theoretical bounds currently rely on $L_\infty$ (maximum) norms, which can be loose; investigating tighter bounds using $L_p$ norms is a planned next step. Finally, the framework currently assumes a shared reward function across tasks. Extending this to settings where both dynamics and rewards vary would further broaden the applicability of the HiP-BMDP framework.

Summary updated

Abstract

Many control tasks exhibit similar dynamics that can be modeled as having common latent structure. Hidden-Parameter Markov Decision Processes (HiP-MDPs) explicitly model this structure to improve sample efficiency in multi-task settings. However, this setting makes strong assumptions on the observability of the state that limit its application in real-world scenarios with rich observation spaces. In this work, we leverage ideas of common structure from the HiP-MDP setting, and extend it to enable robust state abstractions inspired by Block MDPs. We derive instantiations of this new framework for both multi-task reinforcement learning (MTRL) and meta-reinforcement learning (Meta-RL) settings. Further, we provide transfer and generalization bounds based on task and state similarity, along with sample complexity bounds that depend on the aggregate number of samples across tasks, rather than the number of tasks, a significant improvement over prior work. To further demonstrate efficacy of the proposed method, we empirically compare and show improvement over multi-task and meta-reinforcement learning baselines.
BibTeX
@inproceedings{zhang2021learning,
  author = {Amy Zhang and Shagun Sodhani and Khimya Khetarpal and Joelle Pineau},
  title = {{Learning Robust State Abstractions for Hidden-Parameter Block MDPs}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=rhVb_Fo_izc},
}
Figure