Abstract
Control applications often feature tasks with similar, but not identical, dynamics. We introduce the Hidden Parameter Markov Decision Process (HiP-MDP), a framework that parametrizes a family of related dynamical systems with a low-dimensional set of latent factors, and introduce a semiparametric regression approach for learning its structure from data. We show that a learned HiP-MDP rapidly identifies the dynamics of new task instances in several settings, flexibly adapting to task variation.
Introduction
In traditional reinforcement learning (RL), agents typically treat every new environment as a unique problem. For instance, an agent trained to swing a specific baseball bat might fail to generalize when handed a bat of a different length or weight. In reality, many control tasks belong to families governed by the same underlying physics, differing only by specific latent parameters like mass, friction, or length. The challenge lies in enabling agents to transfer knowledge across these tasks and discover these latent parameters without explicit human definitions or kinematic equations.
The HiP-MDP Framework
The authors introduce the Hidden Parameter Markov Decision Process (HiP-MDP) to address these challenges. The framework treats the specific variation of a task as a hidden parameter $\theta_b$ that remains fixed during a single trial. To learn this structure, the model employs a semiparametric regression approach combining Gaussian Processes (GPs) and the Indian Buffet Process (IBP).
Core Components
- Gaussian Processes (GPs): These act as basis functions for the dynamics, modeling state transitions without requiring hard-coded physics equations.
- Indian Buffet Process (IBP): A Bayesian non-parametric tool used to select relevant latent features for specific state dimensions. This allows the model to "discover" the necessary number of latent parameters automatically.
- Latent Weights ($w_{kb}$): These represent the specific values of the hidden parameters for a given task instance.
The learning process consists of two stages: Batch Inference, where the agent learns the global "physics" (GPs and IBP filter) from multiple task instances, and Online Filtering, where the agent quickly estimates the latent weights for a new task instance to enable rapid adaptation.
Experimental Evaluation
The HiP-MDP was evaluated on three classic control benchmarks modified with physical variations:
- Cartpole: Variations in pole mass and length.
- Acrobot: A double-pendulum system with varying segment masses.
- Bicycle: A complex balancing task with variations in height and mass.
Key Insights
- Physics Discovery: The model builds internal representations of physical factors (like gravity or torque) simply by observing behavior across different task instances.
- Latent Mapping: In Cartpole experiments, discovered latent parameters correlated strongly with actual physical mass and length, despite the model having no prior knowledge of these properties.
- Efficiency through Sparsity: The IBP prevents overfitting by only activating latent parameters necessary to explain data variance.
Results
The HiP-MDP demonstrated significant performance gains across all benchmarks. In the Acrobot task, it reached near-optimal performance in just 5 episodes, compared to 15 episodes for an "average model" baseline. In the more complex Bicycle task, the HiP-MDP reached near-optimal performance roughly 150 episodes faster than the average-initialized approach.
Furthermore, in the Cartpole environment, the HiP-MDP significantly outperformed standard regression models in predicting changes in angular velocity, the most critical variable for maintaining balance.
Limitations and Future Work
While effective, the current HiP-MDP framework faces challenges regarding the computational expense of Gaussian Processes as datasets grow. Future research could investigate "drifting" parameters (e.g., a vehicle losing mass as fuel is consumed) and active sensing strategies to help agents more quickly reveal hidden parameters through specific actions.
Summary updated