Introduction
In real-world robotics, agents often operate in Hidden-parameter Markov Decision Processes (HIP-MDPs). These environments contain unobservable physical properties—such as varying mass, friction, or motor scaling—that significantly influence optimal behavior. Standard reinforcement learning (RL) approaches typically struggle with these variations, either treating them as noise or attempting to learn them implicitly through black-box neural networks. This becomes particularly problematic in tasks where the reward function itself is tied to these hidden parameters (e.g., sorting objects by weight).
PrivilegedDreamer addresses this by introducing an explicit module to estimate hidden parameters from history and conditioning the entire agent on these estimates. This approach yields a 41% improvement in average reward over state-of-the-art baselines, especially in complex, parameterized reward scenarios.
Architectural Innovation: Explicit Parameter Estimation
PrivilegedDreamer builds upon the DreamerV2 framework, which utilizes a world model to imagine future trajectories. The core innovation is a dual recurrent architecture designed to handle privileged information explicitly.
The framework consists of four key components:
- Explicit Parameter Estimation: An independent LSTM-based module processes a short history of states and actions to predict hidden parameters ($\omega$).
- Conditioned World Model: The Recurrent State-Space Model (RSSM) is explicitly conditioned on these estimated parameters, enabling physics-accurate "imagination."
- Conditioned Actor-Critic: The policy (actor) and value estimator (critic) directly receive the estimated hidden parameters, allowing for rapid behavioral adaptation as soon as physics are identified.
- Hidden Variable Loss: A specific reconstruction loss forces the latent space to accurately represent hidden physical properties.
Experimental Evaluation
The model was evaluated across five continuous control tasks with randomized physics, including standard DeepMind Control (DMC) tasks like Walker Run and Pendulum Swingup, as well as custom tasks designed to test parameterized rewards.
Two tasks, Kuka Sorting and DMC Pointmass, were particularly challenging. In Kuka Sorting, the robot must move an object to the left if it is heavy and to the right if it is light. Without explicit identification of the object's mass, an agent cannot determine the correct goal, leading to failure in standard RL models.
Key Results and Insights
PrivilegedDreamer outperformed baselines such as DreamerV2, PPO, SAC, and Rapid Motor Adaptation (RMA), achieving a 41% higher average reward across the test suite.
- The Necessity of Explicit Learning: While recurrent networks can theoretically capture hidden information implicitly, the authors found that a dedicated estimation module with a specific loss function is significantly more effective. It forces the model to prioritize system identification early in an episode.
- Solving the Reward Gap: Standard model-based RL often fails when rewards depend on hidden parameters because they lack a structural mechanism to link physical properties to reward-seeking behavior. PrivilegedDreamer's explicit conditioning bridges this gap.
- Sample Efficiency and Rapid Adaptation: The estimation module typically converges within just a few environmental steps. As a model-based approach, it achieves high performance within 2 million timesteps, proving more efficient than many model-free adaptation methods.
Conclusion and Future Directions
PrivilegedDreamer demonstrates that explicit imagination of privileged information is a powerful tool for rapid policy adaptation. While currently focused on state-vector inputs, future work aims to extend this to visual inputs and more complex non-rigid body dynamics. Additionally, the framework holds promise for multi-agent systems where "hidden parameters" could represent the internal intents of other agents.
Summary updated