DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion

Khang Nguyen, An T. Le, Jan Peters, Minh Nhat Vu

TL;DR

This paper introduces DoublyAware, a novel approach for humanoid robot locomotion that tackles uncertainty by explicitly separating planning and policy uncertainties. By combining conformal prediction for robust planning and a group-relative policy constraint for stable learning, DoublyAware achieves faster convergence and more feasible motions compared to existing methods.

What Problem Does This Paper Address?

Humanoid robots are notoriously difficult to control, especially when it comes to locomotion. This is largely due to the inherent uncertainties in their environment and the learning process itself. These uncertainties can be broadly categorized into two types:

  • Aleatoric uncertainty: This is the inherent randomness in the environment, such as unpredictable ground contact or sensor noise. It's like trying to walk on a slippery surface – you can't eliminate the slipperiness itself.
  • Epistemic uncertainty: This arises from the robot's own limited knowledge due to incomplete training. It's like a student who hasn't studied all the material yet – their knowledge is incomplete and can be improved with more learning.

Traditional model-based reinforcement learning (MBRL) methods, particularly those using Temporal Difference Model Predictive Control (TD-MPC), struggle with these dual uncertainties. They often lead to inefficient exploration, unstable learning, and motions that are not physically feasible. This paper aims to address this by explicitly disentangling and addressing these two types of uncertainty in a principled way.

Methods

DoublyAware builds upon the TD-MPC framework but introduces two key innovations:

  • Disjoint Uncertainty Decomposition: The core idea is to treat aleatoric and epistemic uncertainties separately. Aleatoric uncertainty is associated with the planning phase, while epistemic uncertainty is linked to the policy learning phase.
  • Conformal Prediction for Planning Uncertainty: To handle planning uncertainty (aleatoric), DoublyAware uses conformal prediction. This technique allows the robot to filter candidate trajectories generated by its planner (specifically, the MPPI planner) based on statistical guarantees. It ensures that the selected trajectories are "statistically significant" and robust to environmental randomness, effectively providing a risk-bound for planning.
  • Group-Relative Policy Constraint (GRPC) for Policy Uncertainty: To address policy uncertainty (epistemic), DoublyAware incorporates a GRPC optimizer. This method, inspired by recent advancements in large language models, helps stabilize policy learning by imposing an adaptive "trust region" in the latent action space. It encourages the policy to learn from relative action preferences, leading to more stable and data-efficient updates.

By combining these two components, DoublyAware aims to enable the robot to prioritize high-confidence, high-reward behaviors while still exploring effectively and safely.

Datasets

The paper evaluates DoublyAware on the HumanoidBench locomotion suite. This benchmark consists of nine different locomotion tasks designed for humanoid robots. The specific robot used for evaluation is the Unitree 26-DoF H1-2 humanoid.

Key Insights and Ideas

  • The power of disentanglement: Explicitly separating planning and policy uncertainties, rather than treating them as a monolithic problem, is crucial for robust humanoid locomotion. This allows for tailored solutions to each type of uncertainty.
  • Conformal prediction as a safety net for planning: Using conformal prediction to filter trajectories provides a statistically grounded way to ensure robustness against environmental randomness, which is vital for safe exploration.
  • GRPC for stable policy learning: The GRPC approach offers a novel way to stabilize policy updates by focusing on relative action preferences, making learning more data-efficient and less prone to catastrophic failures.
  • Structured uncertainty modeling is key: The paper strongly suggests that simply having a good model is not enough; understanding and explicitly modeling the uncertainty within that model and the environment is paramount for complex robotic tasks.

Key Results

  • Improved Sample Efficiency and Convergence: DoublyAware demonstrates significantly faster convergence and better sample efficiency compared to baseline methods (SAC, BC-SAC, AWAC, TD-MPC2, TD-M(PC)²) on several locomotion tasks, including standing, walking, running, and sitting.
  • Enhanced Motion Feasibility: The generated motions are qualitatively more feasible and coordinated, with the robot exhibiting more natural and stable movements.
  • Performance on Complex Tasks: DoublyAware shows a consistent advantage on tasks requiring whole-body coordination, such as navigating poles, hurdling, and walking on inclined slides.
  • Limitations on Highly Complex Tasks: While performing well on many tasks, DoublyAware, like other methods, struggles with highly complex tasks like crawling and stair-climbing, which require more advanced dynamic skill refinement.
  • Ablation Study Highlights Synergy: An ablation study showed that both conformal prediction (CP) and GRPC contribute significantly, and their combination in DoublyAware yields the best results, suggesting a strong synergistic effect.

Limitations and Future Work

  • Challenging tasks remain: The paper acknowledges that tasks requiring highly advanced whole-body coordination and dynamic skill refinement, such as crawling and stair-climbing, are still challenging for DoublyAware and other methods.
  • Further improvement needed for extreme tasks: While DoublyAware shows robustness, there's still room for improvement in tasks demanding complex full-body movements.
  • Future work could focus on: Extending the method to more complex manipulation tasks, further refining the uncertainty modeling for even more challenging environments, and exploring more advanced policy optimization techniques.

Summary updated

Abstract

Achieving robust robot learning for humanoid locomotion is a fundamental challenge in model-based reinforcement learning (MBRL), where environmental stochasticity and randomness can hinder efficient exploration and learning stability. The environmental, so-called aleatoric, uncertainty can be amplified in high-dimensional action spaces with complex contact dynamics and entangled with epistemic uncertainty in the models during learning phases. In this work, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DoublyAware</i>, an uncertainty-aware extension of Temporal Difference Model Predictive Control (TD-MPC) that explicitly decomposes uncertainty into two disjoint, interpretable components, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i>, planning and policy uncertainties. To handle the planning uncertainty, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DoublyAware</i> employs conformal prediction to filter candidate trajectories using quantile-calibrated risk bounds, ensuring statistical consistency and robustness against stochastic dynamics. Meanwhile, policy rollouts are leveraged as structured informative priors to support the learning phase with Group-Relative Policy Constraint (GRPC) optimizers, which impose a group-based adaptive trust region in the latent action space. This combination enables the robot agent to prioritize high-confidence, high-reward behavior while maintaining effective, targeted exploration under uncertainty. Evaluated on the HumanoidBench locomotion suite with the Unitree 26-DoF H1-2 humanoid, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DoublyAware</i> demonstrates improved sample efficiency, accelerated convergence, and enhanced motion feasibility compared to RL baselines. Our results emphasize the significance of structured uncertainty modeling for data-efficient and reliable decision-making in TD-MPC-based humanoid locomotion learning.
BibTeX
@article{nguyen2025i,
  author = {Khang Nguyen and An T. Le and Jan Peters and Minh Nhat Vu},
  title = {{<i>DoublyAware:</i> Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion}},
  journal = {IEEE Robotics and Automation Letters},
  year = {2025},
  doi = {10.1109/lra.2025.3648611},
}
Figure