Learning Agile Locomotion via Adversarial Training

Yujin Tang, Jie Tan, Tatsuya Harada

TL;DR

Researchers have developed a multi-agent "pursuit-evasion" framework that allows quadruped robots to learn agile locomotion—like sharp turns and reactive maneuvers—without the need for complex, manually designed reward functions. By training a "chaser" robot against an evolving ensemble of "escapee" adversaries, the system automatically discovers high-speed, high-curvature gaits that significantly outperform standard reinforcement learning baselines.

What Problem Does This Paper Address?

While robots have become quite good at walking or running in straight lines, achieving true "agility"—the kind of rapid, coordinated maneuvering seen in hunting animals—remains a massive challenge.

Traditionally, engineers have to meticulously design training environments and reward functions to "nudge" a robot toward agile behavior. If you reward only speed, the robot runs fast but can't turn. If you reward steering, it might become unstable. This manual tuning is laborious and often fails to capture the intricate balance and contact force manipulation required for sharp maneuvers. The authors ask: Can we replace human design effort with a competitive "arms race" between agents to force agility to emerge naturally?

Methods

The authors propose an adversarial learning system using a simulated Unitree Laikago quadruped. The setup is a game of tag:

  1. The Protagonist (Chaser): Learns to catch the adversary.
  2. The Adversary (Escapee): A simplified "dot-bot" that learns to evade the chaser.

Key Technical Choices: * Evolution Strategy (CMA-ES): Instead of standard actor-critic RL (like PPO or SAC), the authors use Covariance Matrix Adaptation Evolution Strategy. They found that simultaneous multi-agent RL algorithms (like MADDPG) struggled to learn both basic locomotion and high-level tactics at the same time. * Iterative Training: They decouple the learning. They train the chaser to beat the current adversaries, then train new adversaries to exploit the chaser’s new weaknesses, and repeat. * Adversary Ensembles: This is the "secret sauce." Instead of the chaser only facing its most recent opponent, it trains against an ensemble of all previous successful adversaries. This prevents "catastrophic forgetting," where the robot learns a new trick but forgets how to handle an old one.

Datasets

The study is conducted entirely in the PyBullet physics simulator. While there isn't a "dataset" in the traditional computer vision sense, the authors evaluated their model against several benchmarks:

  • Manually Designed Baselines: Environments where the target moves in fixed patterns (Cone, Circular, and Zigzag configurations).
  • Sine Trajectories: A test of agility where the chaser must follow a target moving along high-curvature sine waves.
  • Unseen Adversaries: A test set of 8 specialized escaping agents that were held out during the chaser's training to test generalization.

Key Insights and Ideas

  • Competition as a Proxy for Complexity: You don't need to define "agility" in code if you define a "goal" (catching) that requires agility to achieve. The adversary's evolving "dodging" maneuvers naturally force the chaser to learn how to lean into turns and use its legs as pivots.
  • The Necessity of Ensembles: Training against a single adversary leads to overfitting. For example, if an adversary always dodges right, the chaser learns to lean right—becoming easy prey for an adversary that suddenly dodges left. Maintaining a "library" of strategies (the ensemble) forces the chaser to remain versatile.
  • Decoupling Locomotion and Strategy: By using a kinematically controlled "dot-bot" as the adversary, the system ensures the "escapee" can focus entirely on clever maneuvers, which in turn provides a better "teacher" for the complex quadruped chaser.

Key Results

  • Superior Agility: The adversarial-trained policy ($\pi_{adversary}$) achieved a 97% catch rate on complex trajectories, compared to only 61% for the best manually designed baseline.
  • Higher Speeds in Turns: On high-curvature paths, the proposed method maintained higher average speeds (2.40 m/s) and much lower heading errors than baselines, which often fell over or had to slow down significantly to make turns.
  • Failure of Standard MARL: State-of-the-art multi-agent RL algorithms like MADDPG and MATD3 failed to learn stable locomotion at all in this setting, highlighting the difficulty of co-evolving low-level motor control and high-level tactics.

Limitations and Future Work

  • Biological Realism: The learned gaits, while effective, don't look particularly "animal-like." The authors note that their reward function doesn't prioritize energy efficiency, leading to some "robotic" or jittery movements.
  • Sim-to-Real Gap: The work was performed entirely in simulation. Deploying these agile controllers on physical hardware remains a challenge due to sensor noise, latency, and the risk of mechanical damage during high-speed maneuvers.
  • Imitation Learning: The authors suggest that pre-training the robot with motion capture data from real animals before starting the adversarial process could result in more natural and efficient agile locomotion.

Summary updated

Abstract

Developing controllers for agile locomotion is a long-standing challenge for legged robots. Reinforcement learning (RL) and Evolution Strategy (ES) hold the promise of automating the design process of such controllers. However, dedicated and careful human effort is required to design training environments to promote agility. In this paper, we present a multi-agent learning system, in which a quadruped robot (protagonist) learns to chase another robot (adversary) while the latter learns to escape. We find that this adversarial training process not only encourages agile behaviors but also effectively alleviates the laborious environment design effort. In contrast to prior works that used only one adversary, we find that training an ensemble of adversaries, each of which specializes in a different escaping strategy, is essential for the protagonist to master agility. Through extensive experiments, we show that the locomotion controller learned with adversarial training significantly outperforms carefully designed baselines.
BibTeX
@inproceedings{tang2020learning,
  author = {Yujin Tang and Jie Tan and Tatsuya Harada},
  title = {{Learning Agile Locomotion via Adversarial Training}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=w99a25nBnn},
}

Loading related content...

Figure