TL;DR
This paper introduces DoublyAware, a novel approach for humanoid robot locomotion that tackles uncertainty by explicitly separating planning and policy uncertainties. By combining conformal prediction for robust planning and a group-relative policy constraint for stable learning, DoublyAware achieves faster convergence and more feasible motions compared to existing methods.
What Problem Does This Paper Address?
Humanoid robots are notoriously difficult to control, especially when it comes to locomotion. This is largely due to the inherent uncertainties in their environment and the learning process itself. These uncertainties can be broadly categorized into two types:
- Aleatoric uncertainty: This is the inherent randomness in the environment, such as unpredictable ground contact or sensor noise. It's like trying to walk on a slippery surface – you can't eliminate the slipperiness itself.
- Epistemic uncertainty: This arises from the robot's own limited knowledge due to incomplete training. It's like a student who hasn't studied all the material yet – their knowledge is incomplete and can be improved with more learning.
Traditional model-based reinforcement learning (MBRL) methods, particularly those using Temporal Difference Model Predictive Control (TD-MPC), struggle with these dual uncertainties. They often lead to inefficient exploration, unstable learning, and motions that are not physically feasible. This paper aims to address this by explicitly disentangling and addressing these two types of uncertainty in a principled way.
Methods
DoublyAware builds upon the TD-MPC framework but introduces two key innovations:
- Disjoint Uncertainty Decomposition: The core idea is to treat aleatoric and epistemic uncertainties separately. Aleatoric uncertainty is associated with the planning phase, while epistemic uncertainty is linked to the policy learning phase.
- Conformal Prediction for Planning Uncertainty: To handle planning uncertainty (aleatoric), DoublyAware uses conformal prediction. This technique allows the robot to filter candidate trajectories generated by its planner (specifically, the MPPI planner) based on statistical guarantees. It ensures that the selected trajectories are "statistically significant" and robust to environmental randomness, effectively providing a risk-bound for planning.
- Group-Relative Policy Constraint (GRPC) for Policy Uncertainty: To address policy uncertainty (epistemic), DoublyAware incorporates a GRPC optimizer. This method, inspired by recent advancements in large language models, helps stabilize policy learning by imposing an adaptive "trust region" in the latent action space. It encourages the policy to learn from relative action preferences, leading to more stable and data-efficient updates.
By combining these two components, DoublyAware aims to enable the robot to prioritize high-confidence, high-reward behaviors while still exploring effectively and safely.
Datasets
The paper evaluates DoublyAware on the HumanoidBench locomotion suite. This benchmark consists of nine different locomotion tasks designed for humanoid robots. The specific robot used for evaluation is the Unitree 26-DoF H1-2 humanoid.
Key Insights and Ideas
- The power of disentanglement: Explicitly separating planning and policy uncertainties, rather than treating them as a monolithic problem, is crucial for robust humanoid locomotion. This allows for tailored solutions to each type of uncertainty.
- Conformal prediction as a safety net for planning: Using conformal prediction to filter trajectories provides a statistically grounded way to ensure robustness against environmental randomness, which is vital for safe exploration.
- GRPC for stable policy learning: The GRPC approach offers a novel way to stabilize policy updates by focusing on relative action preferences, making learning more data-efficient and less prone to catastrophic failures.
- Structured uncertainty modeling is key: The paper strongly suggests that simply having a good model is not enough; understanding and explicitly modeling the uncertainty within that model and the environment is paramount for complex robotic tasks.
Key Results
- Improved Sample Efficiency and Convergence: DoublyAware demonstrates significantly faster convergence and better sample efficiency compared to baseline methods (SAC, BC-SAC, AWAC, TD-MPC2, TD-M(PC)²) on several locomotion tasks, including standing, walking, running, and sitting.
- Enhanced Motion Feasibility: The generated motions are qualitatively more feasible and coordinated, with the robot exhibiting more natural and stable movements.
- Performance on Complex Tasks: DoublyAware shows a consistent advantage on tasks requiring whole-body coordination, such as navigating poles, hurdling, and walking on inclined slides.
- Limitations on Highly Complex Tasks: While performing well on many tasks, DoublyAware, like other methods, struggles with highly complex tasks like crawling and stair-climbing, which require more advanced dynamic skill refinement.
- Ablation Study Highlights Synergy: An ablation study showed that both conformal prediction (CP) and GRPC contribute significantly, and their combination in DoublyAware yields the best results, suggesting a strong synergistic effect.
Limitations and Future Work
- Challenging tasks remain: The paper acknowledges that tasks requiring highly advanced whole-body coordination and dynamic skill refinement, such as crawling and stair-climbing, are still challenging for DoublyAware and other methods.
- Further improvement needed for extreme tasks: While DoublyAware shows robustness, there's still room for improvement in tasks demanding complex full-body movements.
- Future work could focus on: Extending the method to more complex manipulation tasks, further refining the uncertainty modeling for even more challenging environments, and exploring more advanced policy optimization techniques.
Summary updated