TL;DR
This paper introduces a novel two-stage learning framework for quadrupedal robots to master highly dynamic acrobatic motions like jumping, pronking, and back-flipping, even for robots with complex articulated soft bodies. By combining a gradient-free evolution strategy for initial policy discovery with deep reinforcement learning for refinement, the method bypasses the need for expert demonstrations and simplifies reward shaping, enabling robots to learn these challenging maneuvers from scratch.
What Problem Does This Paper Address?
Quadrupedal robots are increasingly being explored for their mobility and versatility. While basic gaits are relatively easy to learn, achieving highly dynamic and acrobatic motions (like jumping or back-flipping) poses significant challenges. These difficulties stem from the robots' under-actuated, hybrid, and nonlinear dynamics. Traditional methods like trajectory optimization (TO) and model predictive control (MPC) are computationally intensive and often require accurate models, leading to conservative solutions. Reinforcement learning (RL) offers a promising alternative, but learning complex, sparse-reward tasks like acrobatic maneuvers typically requires extensive reward shaping or reliance on pre-existing reference motions or controllers. This reliance on expert knowledge limits the applicability of existing RL methods. Furthermore, the growing trend of incorporating soft, articulated bodies into robots, while beneficial for compliance and adaptability, introduces even greater control complexities that are not well-addressed by current dynamic motion generation techniques.
Methods
The paper proposes a two-stage learning framework:
- Stage 1: Gradient-Free Evolution Strategy (ES) for Initial Policy Discovery:
- The authors employ the Augmented Random Search (ARS) algorithm, a gradient-free ES method.
- This stage treats the RL task as a black-box optimization problem.
- Crucially, it learns a simple linear policy (action = W * state + b). This simplicity eliminates the need for predefined reference motions and significantly reduces the burden of reward shaping, as ES can handle sparse rewards effectively.
- Stage 2: Deep Reinforcement Learning (DRL) for Policy Refinement:
- The linear policy learned in Stage 1 is transferred to a more expressive neural network (NN).
- Proximal Policy Optimization (PPO) is used to fine-tune this NN policy.
- The DRL training is "warm-started" by mimicking the motions generated by the Stage 1 policy, using an imitative reward function.
- Subsequently, the policy is further refined using task-specific rewards. This two-stage approach leverages the strengths of both ES (simplicity, sparse reward handling) and DRL (expressiveness, fine-tuning).
A behavior selector mechanism is also introduced for acrobatic motions, which transitions through distinct take-off, flight, and landing phases, each with specific control strategies. For instance, the flight phase often uses a homing pose, and the landing phase employs a PD controller for compliance.
Datasets
Not discussed in the paper. The experiments are conducted in simulation using the PyBullet environment.
Key Insights and Ideas
- Two-stage learning is powerful: Combining a simple, gradient-free initial learner (ES) with a powerful, expressive learner (DRL) is highly effective for complex, sparse-reward tasks. The ES stage effectively bootstraps the DRL stage, overcoming initial exploration challenges.
- Linear policies can be surprisingly effective: The paper demonstrates that a simple linear policy learned via ES can achieve competitive performance, even outperforming some NN-based policies in the initial stage. This highlights the efficiency of ES for discovering fundamental control strategies.
- Reduced reliance on expert knowledge: By learning from scratch without reference motions, the method significantly broadens the applicability of dynamic motion generation to new robots and tasks where expert data might be scarce or unavailable.
- Simplified reward shaping: The ES stage's ability to handle sparse rewards alleviates the labor-intensive process of designing dense reward functions, a common bottleneck in RL for dynamic tasks.
- Soft robots benefit from dynamic learning: The framework is shown to be particularly effective for articulated soft quadrupeds, enabling them to exploit their inherent compliance for enhanced dynamic performance, a capability not fully realized by traditional methods.
Key Results
- The proposed two-stage learning framework successfully enabled quadrupedal robots to learn highly dynamic motions, including jumping (in place and forward), pronking, and back-flipping, from scratch.
- For the jumping task, the articulated soft quadruped achieved 15.4% higher jumps and 23.1% further jumps compared to its rigid counterpart, demonstrating the advantage of exploiting parallel elasticity.
- The "ARS+DRL" approach consistently achieved higher rewards and better performance metrics (like jumping height and distance) compared to the "ARS" (ES only) baseline.
- The linear policy in the first stage (ARS) showed faster convergence and a smoother reward profile than some NN-based policies in the same stage.
- The method proved robust to environment randomization, with the retrained policies ("ARS-r", "ARS+DRL-r") showing improved success rates in simulated randomized environments.
Limitations and Future Work
- Hardware Validation: The authors acknowledge that future work will focus on hardware validation to confirm the sim-to-real transferability of the learned policies.
- Generalization to Humanoid Robots: The authors express interest in applying the method to humanoid robots and generating more general locomotion control policies.
- Exploration of Other Dynamic Motions: While pronking and back-flipping were demonstrated, further exploration of other complex dynamic motions could be a future direction.
- Hyperparameter Tuning: While the paper provides hyperparameters, the tuning process for such complex tasks can still be a challenge.
Summary updated