Efficient Reinforcement Learning for Jumping Monopods

Riccardo Bussola, Michele Focchi, Andrea Del Prete, Daniele Fontanelli, Luigi Palopoli

Abstract: Precise jumping on uneven terrain is a critical capability for legged robots, yet traditional trajectory optimization (TO) methods are often too slow for real-time use, while end-to-end reinforcement learning (RL) suffers from poor sample efficiency. This paper presents Guided Reinforcement Learning (GRL), an approach that constrains an RL agent to output parameters for Cartesian Bezier curves. This hybrid framework achieves sub-millisecond inference times and superior accuracy compared to optimization-based baselines, particularly in challenging "back jump" scenarios, while requiring significantly fewer training episodes than end-to-end alternatives.

Introduction

Jumping represents a high-stakes maneuver for legged robots. Unlike walking, which allows for continuous feedback corrections, a jump's success is largely determined during the explosive thrust phase. Minor deviations at lift-off can lead to substantial landing errors. Existing solutions typically rely on Trajectory Optimization (TO) or end-to-end Reinforcement Learning (RL). TO methods provide high accuracy but are computationally intensive, often requiring seconds to solve—far too slow for real-time reactive adjustments. Conversely, end-to-end RL agents often struggle to converge on stable solutions for complex jumping tasks, requiring millions of training episodes. This work proposes Guided Reinforcement Learning (GRL) as a middle ground that is fast, robust, and sample-efficient.

Methods

The GRL framework integrates physical priors into the learning process to "guide" the agent. Instead of direct joint torque control, the RL agent acts as a high-level planner, outputting parameters for a 3rd-order Bezier curve in Cartesian space. This restriction ensures smooth, physically plausible trajectories and drastically reduces the search space.

The system utilizes the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The action space is reduced to five key parameters: thrust time ($T_{th}$), lift-off radius ($r$), and three directional angles ($\theta, r_v, \theta_v$). As illustrated in the system architecture, these parameters are processed by a Bezier planner and then translated into joint torques via a low-level PD controller with gravity compensation.

System architecture showing the RL Agent as a high-level planner for the Bezier-based trajectory.

Experimental Setup

The approach was evaluated in the Gazebo simulation environment. The training region consisted of targets randomly sampled within a cylindrical volume (radius 0.65m, height 0.25m to 0.5m). To test generalization, the agent was evaluated on a test region 20% larger than the training area, comprising 726 uniformly spaced target positions.

Results and Discussion

A key finding is the agent's ability to learn compensations for low-level controller inaccuracies. By observing the relationship between its Cartesian actions and landing errors, the RL agent learns to "distort" the Bezier curve to counteract physical effects like friction or lag.

GRL demonstrated significant advantages over both TO (using FDDP) and end-to-end RL:

  • Inference Speed: GRL achieves an inference time of 0.7 ms, compared to 7.6–17 seconds for FDDP, enabling real-time operation.
  • Accuracy and Reach: GRL achieved a Relative Percentual Error (RPE) of ~16% for front jumps. Notably, it outperformed optimization on "back jumps," where TO solvers frequently failed (52% error vs. GRL's significantly lower error).
  • Sample Efficiency: While end-to-end RL failed to converge after 1 million steps, GRL reached satisfactory performance in just 50,000 episodes.
Top-view of the feasibility region showing how the agent's reach expands and improves with more training episodes compared to the optimization baseline.

Conclusion

By combining the flexibility of RL with the structure of Bezier-based trajectory planning, GRL provides a robust solution for monopod jumping. Future work will address the current model's limitations, such as the lack of angular momentum handling, and extend the framework to full quadruped robots capable of complex 3D maneuvers like flips and twists.

Summary updated

BibTeX
@article{bussola2024efficient,
  author = {Riccardo Bussola and Michele Focchi and Andrea Del Prete and Daniele Fontanelli and Luigi Palopoli},
  title = {{Efficient Reinforcement Learning for Jumping Monopods}},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24154981},
}
Figure