Learning Agile Locomotion via Adversarial Training

Yujin Tang, Jie Tan, Tatsuya Harada

Abstract: Developing controllers for agile locomotion is a long-standing challenge for legged robots. Reinforcement learning (RL) and Evolution Strategy (ES) hold the promise of automating the design process of such controllers. However, dedicated and careful human effort is required to design training environments to promote agility. In this paper, we present a multi-agent learning system, in which a quadruped robot (protagonist) learns to chase another robot (adversary) while the latter learns to escape. We find that this adversarial training process not only encourages agile behaviors but also effectively alleviates the laborious environment design effort. In contrast to prior works that used only one adversary, we find that training an ensemble of adversaries, each of which specializes in a different escaping strategy, is essential for the protagonist to master agility. Through extensive experiments, we show that the locomotion controller learned with adversarial training significantly outperforms carefully designed baselines.

Introduction

While legged robots have become proficient at walking and running in straight lines, achieving true "agility"—the rapid, coordinated maneuvering seen in hunting animals—remains a significant hurdle. Traditionally, engineers must meticulously design training environments and reward functions to nudge a robot toward agile behavior. This manual tuning is laborious and often fails to capture the intricate balance and contact force manipulation required for sharp maneuvers. This research explores whether a competitive "arms race" between agents can replace human design effort, allowing agility to emerge naturally.

Adversarial Learning Framework

The proposed system utilizes a simulated Unitree Laikago quadruped in a multi-agent "game of tag." The setup involves a Protagonist (Chaser) that learns to catch an Adversary (Escapee). To ensure the adversary provides a high-quality challenge, it is modeled as a kinematically controlled "dot-bot" that focuses entirely on clever maneuvers.

The pursuit-evasion task: a quadruped robot (protagonist) learns to chase a red sphere (adversary) through high-curvature trajectories.

Key Technical Choices

  • Evolution Strategy (CMA-ES): The authors utilize Covariance Matrix Adaptation Evolution Strategy rather than standard actor-critic RL, finding it more stable for co-evolving low-level motor control and high-level tactics.
  • Iterative Training and Ensembles: To prevent "catastrophic forgetting," the chaser is trained against an evolving ensemble of all previous successful adversaries. This forces the chaser to remain versatile rather than overfitting to a single opponent's strategy.

Experimental Setup and Baselines

The study was conducted in the PyBullet physics simulator. To evaluate the effectiveness of adversarial training, the model was tested against several manually designed benchmarks where targets moved in fixed patterns.

Manually designed baseline configurations (Cone, Circular, and Zigzag) used to evaluate the chaser's performance against standard RL training environments.

Beyond these configurations, the chaser was tested on high-curvature sine trajectories and a held-out set of eight specialized escaping agents to verify generalization capabilities.

Key Results and Insights

  • Superior Agility: The adversarial-trained policy achieved a 97% catch rate on complex trajectories, compared to only 61% for the best manually designed baseline.
  • High-Speed Maneuvering: On high-curvature paths, the proposed method maintained an average speed of 2.40 m/s with minimal heading errors. In contrast, baseline controllers often fell over or slowed down significantly to navigate turns.
  • Competition as a Proxy for Complexity: The results suggest that defining a goal (catching) that requires agility is more effective than trying to define "agility" itself in code. The adversary's evolving maneuvers naturally force the chaser to learn how to lean into turns and use its legs as pivots.

Conclusion and Future Work

While the adversarial framework successfully automates the discovery of agile gaits, the resulting movements can appear jittery as the reward function does not prioritize energy efficiency. Future research will focus on improving biological realism—perhaps through imitation learning from animal motion capture—and addressing the sim-to-real gap to deploy these controllers on physical hardware.

Summary updated

Abstract

Developing controllers for agile locomotion is a long-standing challenge for legged robots. Reinforcement learning (RL) and Evolution Strategy (ES) hold the promise of automating the design process of such controllers. However, dedicated and careful human effort is required to design training environments to promote agility. In this paper, we present a multi-agent learning system, in which a quadruped robot (protagonist) learns to chase another robot (adversary) while the latter learns to escape. We find that this adversarial training process not only encourages agile behaviors but also effectively alleviates the laborious environment design effort. In contrast to prior works that used only one adversary, we find that training an ensemble of adversaries, each of which specializes in a different escaping strategy, is essential for the protagonist to master agility. Through extensive experiments, we show that the locomotion controller learned with adversarial training significantly outperforms carefully designed baselines.
BibTeX
@inproceedings{tang2020learning,
  author = {Yujin Tang and Jie Tan and Tatsuya Harada},
  title = {{Learning Agile Locomotion via Adversarial Training}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=w99a25nBnn},
}
Figure