Abstract
Controlling a non-statically bipedal robot is challenging due to the complex dynamics and multi-criterion optimization involved. Recent works have demonstrated the effectiveness of deep reinforcement learning (DRL) for simulation and physical robots. In these methods, the rewards from different criteria are normally summed to learn a single value function. However, this may cause the loss of dependency information between hybrid rewards and lead to a sub-optimal policy. In this work, we propose a novel reward-adaptive reinforcement learning for biped locomotion, allowing the control policy to be simultaneously optimized by multiple criteria using a dynamic mechanism. The proposed method applies a multi-head critic to learn a separate value function for each reward component. This leads to hybrid policy gradient. We further propose dynamic weight, allowing each component to optimize the policy with different priorities. This hybrid and dynamic policy gradient (HDPG) design makes the agent learn more efficiently. We show that the proposed method outperforms summed-up-reward approaches and is able to transfer to physical robots. The sim-to-real and MuJoCo results further demonstrate the effectiveness and generalization of HDPG.
Introduction
Teaching a two-legged robot to walk is a classic multi-criterion optimization problem. An agent must simultaneously maintain balance, coordinate limbs, minimize energy consumption, and move forward. Standard Deep Reinforcement Learning (DRL) typically handles this by summing all these different goals into a single scalar reward. However, this "reward-summing" approach often discards the dependency information between different criteria. If a robot receives a low total reward, it lacks clarity on whether it failed due to balance issues or energy inefficiency. This lack of granular feedback often leads to sub-optimal policies and slow training.
To address this, researchers have developed Hybrid and Dynamic Policy Gradient (HDPG). This approach masters bipedal locomotion by breaking down complex rewards into individual components and dynamically prioritizing tasks based on their relative difficulty during training.
Methodology
HDPG builds upon the Deep Deterministic Policy Gradient (DDPG) framework with two novel architectural changes designed to improve learning efficiency and policy robustness.
Multi-head Critic (Hybrid Component)
Instead of a single critic network predicting a total value, the HDPG critic utilizes multiple "heads." Each head learns a separate value function for a specific reward component, such as gait, orientation, or torque. This architecture prevents different reward signals from washing each other out during backpropagation, maintaining the correlation between specific actions and specific outcomes.
Dynamic Policy Gradient (Dynamic Component)
Not all rewards are equally easy to earn. HDPG introduces a dynamic weighting mechanism that adjusts the priority of each reward component during training. By analyzing the mean and variance of rewards, the system automatically prioritizes "simpler" components (like maintaining height) before tackling "challenging" ones (like forward velocity). This creates an automated curriculum for the agent.
The reward function is decomposed into six sub-rewards: Gait, Step (distance), Torque (efficiency), Height, Orientation, and Fall Down (penalty).
Experiments and Results
The researchers evaluated HDPG using a custom-built Gazebo environment featuring the AIDA robot, a 10-degree-of-freedom bipedal platform. The algorithm was tested against several benchmarks, including push recovery (resisting forces from 6N to 14N), obstacle crossing, and slope walking.
Key Findings
- Learning Efficiency: In simulation, HDPG's cumulative reward began to climb after just 2k episodes, whereas the standard DDPG baseline required 6k episodes to show similar progress.
- Robustness: In an 8N sideward push challenge, HDPG maintained a 90%+ success rate, outperforming both standard DDPG and static multi-head versions. Traditional control methods (ZMP) failed significantly more often as force increased.
- Generalization: When applied to standard OpenAI Gym (MuJoCo) benchmarks like Hopper-v3 and Walker2d-v3, HDPG (and its PPO-based variant, HD-PPO) consistently outperformed baselines, achieving up to a 25% performance breakthrough.
- Sim-to-Real Transfer: By using dynamics randomization in the simulator, the HDPG policy transferred successfully to a physical bipedal robot, allowing it to recover from physical shoves in real-time.
Discussion and Limitations
While HDPG demonstrates superior learning speed and robustness, the initial selection and decomposition of reward components still require manual human engineering. Future work aims to automate this decomposition and test the robot on more complex real-world terrains, such as undulating ground and varied obstacles. Additionally, while the multi-head architecture improves performance, it increases the number of parameters in the critic, leading to higher computational overhead during the training phase.
Summary updated