HACL: History-Aware Curriculum Learning for Fast Locomotion

Prakhar Mishra, Amir Hossain Raj, Xuesu Xiao, Dinesh ManochaarXiv 2025 · DOI · arXiv

Abstract

Achieving high-speed locomotion in quadrupedal robots remains a significant challenge in reinforcement learning (RL). Traditional curriculum learning methods often struggle with instability at high velocities and fail to account for the non-Markovian nature of the training process. This paper presents History-Aware Curriculum Learning (HACL), a meta-scheduling algorithm that explicitly models the history of training rewards using a recurrent neural network (RNN). HACL enables robots to achieve record-breaking speeds—6.7 m/s in simulation and 4.1 m/s on real hardware—while maintaining superior stability and efficiency compared to existing baselines.

Introduction

Training legged robots for high-speed sprinting typically involves curriculum learning, where task difficulty increases as the robot's performance improves. However, current approaches frequently encounter two major hurdles. First, aggressive advancement can lead to "myopic" policies that prioritize immediate rewards over long-term stability. Second, most curricula treat training as a Markovian process, ignoring the temporal dependencies inherent in robot locomotion, such as sensor latency and varying environmental friction. These factors make locomotion a Partially Observable Markov Decision Process (POMDP), requiring a history-aware approach for effective learning.

Methodology: History-Aware Curriculum Learning (HACL)

HACL introduces a decoupled architecture where a meta-scheduler manages task difficulty independently of the low-level control policy.

The Meta-Policy

The core of HACL is a meta-policy ($\pi_{meta}$) implemented as an RNN (LSTM or GRU). Unlike standard schedulers that only consider current performance, the HACL meta-policy tracks the history of rewards across training episodes. It predicts expected returns for a discretized 3D grid of 4,000 command velocity bins (linear $v_x, v_y$, and angular $\omega_z$).

The HACL framework architecture, showing the interaction between the meta-scheduler, the environment, and the low-level policy.

HA-Greedy Sampling

To optimize exploration, HACL employs "History-Aware Greedy" sampling. This rule weights velocity bins based on predicted returns, focusing training on high-value regions. This prevents the robot from becoming trapped in low-reward local optima and ensures a stable progression toward higher speeds.

Training and Deployment

The low-level policy ($\pi_{\theta}$) is trained using Proximal Policy Optimization (PPO) to execute joint commands. To bridge the sim-to-real gap, a teacher-student training paradigm is used, where a student policy learns to replicate a privileged teacher policy using only onboard sensor history.

Experimental Results

The HACL framework was validated in Isaac Gym using models of the MIT Mini Cheetah and Unitree Go1/Go2 robots, followed by real-world deployment on a Unitree Go1 EDU.

Performance Benchmarks

HACL achieved a peak simulation speed of 6.7 ± 0.2 m/s, significantly surpassing previous state-of-the-art methods (approx. 5.7 m/s). In real-world testing, the robot reached 4.1 ± 0.2 m/s, one of the highest reported speeds for its hardware class.

Learning curves comparing HACL against various velocity targets, demonstrating consistent convergence to high speeds.

Robustness and Stability

Beyond raw speed, HACL-trained policies exhibited high stability. The robot maintained a 90% success rate on woolen carpets and cement slopes, and a 60% success rate on challenging pebble terrain. It demonstrated the ability to recover from trips and bumps within milliseconds, returning to a stable gait without manual intervention.

Discussion and Conclusion

The success of HACL underscores the importance of history in curriculum learning for complex robotic tasks. By utilizing an RNN-based meta-scheduler, the system avoids the instability and "forgetting" issues common in simpler MLP-based curricula.

Despite these gains, a gap remains between simulation and real-world performance (6.7 m/s vs. 4.1 m/s), likely due to unmodeled actuator latency and thermal limits of the hardware. Future research will focus on narrowing this gap and extending the HACL framework to diverse robot morphologies, including bipeds and humanoids.

Summary updated

Abstract

We address the problem of agile and rapid locomotion, a key characteristic of quadrupedal and bipedal robots. We present a new algorithm that maintains stability and generates high-speed trajectories by considering the temporal aspect of locomotion. Our formulation takes into account past information based on a novel history-aware curriculum Learning (HACL) algorithm. We model the history of joint velocity commands with respect to the observed linear and angular rewards using a recurrent neural net (RNN). The hidden state helps the curriculum learn the relationship between the forward linear velocity and angular velocity commands and the rewards over a given time-step. We validate our approach on the MIT Mini Cheetah,Unitree Go1, and Go2 robots in a simulated environment and on a Unitree Go1 robot in real-world scenarios. In practice, HACL achieves peak forward velocity of 6.7 m/s for a given command velocity of 7m/s and outperforms prior locomotion algorithms by nearly 20%.
BibTeX
@misc{mishra2025hacl,
  author = {Prakhar Mishra and Amir Hossain Raj and Xuesu Xiao and Dinesh Manocha},
  title = {{HACL: History-Aware Curriculum Learning for Fast Locomotion}},
  year = {2025},
  eprint = {2505.18429},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18429},
}
Figure