TL;DR
This paper introduces a "Guided Reinforcement Learning" (GRL) framework that significantly speeds up the learning process for a monopod robot to perform accurate 3D jumps. By incorporating nature-inspired heuristics and a structured approach to the action space, GRL achieves high accuracy and safety with drastically reduced training time compared to traditional optimization and end-to-end RL methods.
What Problem Does This Paper Address?
Jumping is a complex maneuver for legged robots, requiring precise control over a short but critical thrust phase to achieve a desired landing location and orientation. Traditional methods like non-linear optimization are computationally expensive, making real-time applications challenging. End-to-end Reinforcement Learning (RL) approaches, while promising, often struggle with the sparse rewards and non-smooth dynamics inherent in jumping tasks, leading to prohibitively long training times and suboptimal performance. This paper addresses the need for an efficient, accurate, and safe RL-based solution for omnidirectional 3D jumping that can operate in real-time and learn quickly.
Methods
The core of the proposed solution is the Guided Reinforcement Learning (GRL) framework. Key aspects include:
- Cartesian Space Learning: The RL agent learns in Cartesian space rather than joint space. This allows for better generalization across different robot morphologies and enables a simple safety filter.
- Heuristic Trajectory Shaping: Instead of learning the entire motion from scratch, GRL leverages a heuristic inspired by biological jumps. The thrust phase trajectory (from standstill to lift-off) is parameterized using a 3rd-order Bézier curve. This curve is defined by four control points, with the first and last points fixed to the initial and lift-off CoM positions, respectively. The RL agent learns the parameters of this Bézier curve.
- Reduced Action Space: By expressing the lift-off CoM position and velocity in spherical coordinates and considering the jump plane, the action space is reduced from 7 to 5 dimensions (radius, yaw, pitch, radial velocity, pitch velocity, and thrust duration). Domain knowledge is further used to bound these parameters, preventing physically impossible actions.
- Physically Informative Reward Function: The reward function is designed to penalize violations of physical constraints (e.g., joint limits, friction, singularity) and reward landing close to the target. This "reward shaping" guides the learning process effectively.
- Low-Level Controller: A standard PD controller with gravity compensation tracks the Bézier curve references generated by the RL agent.
Datasets
The paper does not explicitly mention or introduce new datasets. Instead, it relies on a simulated environment (Gazebo) for training and evaluating the RL agent. The simulation parameters for the robot, controller, and environment are detailed in Table 1. The evaluation involves generating targets within a predefined training region and testing generalization on a larger test region.
Key Insights and Ideas
- Heuristics are Crucial for Efficiency: Injecting prior knowledge, like the Bézier curve structure for jump trajectories, drastically reduces the search space and speeds up RL learning, making it feasible for complex tasks like jumping.
- Cartesian Space Learning Offers Advantages: Learning in Cartesian space, rather than joint space, simplifies the problem and allows for easier implementation of safety checks and better generalization.
- Action Space Design Matters: Carefully designing and constraining the action space based on physical intuition is key to efficient learning and preventing the agent from exploring physically impossible or undesirable motions.
- Robustness to Tracking Errors: The GRL approach demonstrates robustness to small tracking inaccuracies from the low-level controller, a significant advantage over methods that assume perfect execution.
Key Results
- Drastic Reduction in Training Time: GRL achieved good learning performance in a few thousand training episodes, significantly outperforming end-to-end RL approaches which required millions of steps and still did not converge.
- High Accuracy and Safety: The GRL agent achieved high accuracy in reaching target locations, with a Relative Percentual Error (RPE) of approximately 16% for front jumps after 100k episodes. Feasibility constraints were largely satisfied after 10k episodes.
- Superior to Optimization for Back Jumps: GRL outperformed a standard Non-Linear Trajectory Optimization (FDDP) method, particularly for backward jumps, where FDDP struggled due to asymmetry.
- Real-time Capability: The computation time for the GRL approach is below a few milliseconds, enabling real-time execution at controller rates (kHz).
- Generalization: The trained policy demonstrated good generalization capabilities, successfully executing jumps in a test region 20% larger than the training region.
Limitations and Future Work
- Limitations:
- The current work focuses on a monopod robot and primarily on translational jumps.
- The landing phase is handled by a separate controller, not learned by the GRL agent.
- The current approach assumes a simplified robot model (3 DoF monopod with passive prismatic joints).
- Future Work:
- Extend the approach to full quadruped robots, including angular motions for more complex jumps (twists, somersaults).
- Integrate a learned landing strategy.
- Improve robustness by including robot non-idealities in the learning phase.
- Further speed up training using parallel computation.
- Develop a jump reflex that can be triggered even when the robot is already in motion (non-zero initial velocity).
Summary updated