Abstract
Achieving high-speed locomotion in quadrupedal robots remains a significant challenge in reinforcement learning (RL). Traditional curriculum learning methods often struggle with instability at high velocities and fail to account for the non-Markovian nature of the training process. This paper presents History-Aware Curriculum Learning (HACL), a meta-scheduling algorithm that explicitly models the history of training rewards using a recurrent neural network (RNN). HACL enables robots to achieve record-breaking speeds—6.7 m/s in simulation and 4.1 m/s on real hardware—while maintaining superior stability and efficiency compared to existing baselines.
Introduction
Training legged robots for high-speed sprinting typically involves curriculum learning, where task difficulty increases as the robot's performance improves. However, current approaches frequently encounter two major hurdles. First, aggressive advancement can lead to "myopic" policies that prioritize immediate rewards over long-term stability. Second, most curricula treat training as a Markovian process, ignoring the temporal dependencies inherent in robot locomotion, such as sensor latency and varying environmental friction. These factors make locomotion a Partially Observable Markov Decision Process (POMDP), requiring a history-aware approach for effective learning.
Methodology: History-Aware Curriculum Learning (HACL)
HACL introduces a decoupled architecture where a meta-scheduler manages task difficulty independently of the low-level control policy.
The Meta-Policy
The core of HACL is a meta-policy ($\pi_{meta}$) implemented as an RNN (LSTM or GRU). Unlike standard schedulers that only consider current performance, the HACL meta-policy tracks the history of rewards across training episodes. It predicts expected returns for a discretized 3D grid of 4,000 command velocity bins (linear $v_x, v_y$, and angular $\omega_z$).
HA-Greedy Sampling
To optimize exploration, HACL employs "History-Aware Greedy" sampling. This rule weights velocity bins based on predicted returns, focusing training on high-value regions. This prevents the robot from becoming trapped in low-reward local optima and ensures a stable progression toward higher speeds.
Training and Deployment
The low-level policy ($\pi_{\theta}$) is trained using Proximal Policy Optimization (PPO) to execute joint commands. To bridge the sim-to-real gap, a teacher-student training paradigm is used, where a student policy learns to replicate a privileged teacher policy using only onboard sensor history.
Experimental Results
The HACL framework was validated in Isaac Gym using models of the MIT Mini Cheetah and Unitree Go1/Go2 robots, followed by real-world deployment on a Unitree Go1 EDU.
Performance Benchmarks
HACL achieved a peak simulation speed of 6.7 ± 0.2 m/s, significantly surpassing previous state-of-the-art methods (approx. 5.7 m/s). In real-world testing, the robot reached 4.1 ± 0.2 m/s, one of the highest reported speeds for its hardware class.
Robustness and Stability
Beyond raw speed, HACL-trained policies exhibited high stability. The robot maintained a 90% success rate on woolen carpets and cement slopes, and a 60% success rate on challenging pebble terrain. It demonstrated the ability to recover from trips and bumps within milliseconds, returning to a stable gait without manual intervention.
Discussion and Conclusion
The success of HACL underscores the importance of history in curriculum learning for complex robotic tasks. By utilizing an RNN-based meta-scheduler, the system avoids the instability and "forgetting" issues common in simpler MLP-based curricula.
Despite these gains, a gap remains between simulation and real-world performance (6.7 m/s vs. 4.1 m/s), likely due to unmodeled actuator latency and thermal limits of the hardware. Future research will focus on narrowing this gap and extending the HACL framework to diverse robot morphologies, including bipeds and humanoids.
Summary updated