Learning Symmetry and Low-energy Locomotion

Wenhao Yu, Greg Turk, C. Karen Liu

TL;DR

This paper presents a novel deep reinforcement learning (DRL) approach for generating realistic, symmetric, and energy-efficient locomotion for virtual characters. By introducing a "mirror symmetry loss" and a "curriculum learning" method with virtual assistance, the authors achieve natural-looking gaits across diverse character morphologies without relying on motion capture data.

What Problem Does This Paper Address?

Creating lifelike locomotion for animated characters has traditionally been a labor-intensive process, often relying on motion capture data, complex finite state machines, or hand-crafted, morphology-specific rules. While Deep Reinforcement Learning (DRL) has shown promise in automating this process, existing DRL methods often produce jerky, unnatural motions that lack biological plausibility. Specifically, these learned gaits tend to be overly energetic and can exhibit undesirable asymmetries, even in characters with symmetrical bodies. This paper aims to bridge this gap by developing a minimalist DRL approach that can learn natural locomotion behaviors from fundamental principles, without requiring explicit examples of desired motion.

Methods

The authors propose two key modifications to standard DRL for locomotion:

  • Mirror Symmetry Loss: Instead of penalizing state asymmetry, which can be difficult to define and enforce for complex motions, they introduce a loss function that directly penalizes asymmetric actions between paired limbs (e.g., left vs. right leg). This is expressed as $\pi_\theta(s) = \Psi_\alpha(\pi_\theta(\Psi_\phi(s)))$, where $\Psi$ denotes mirroring operations. This loss is incorporated into the overall objective function, allowing for the learning of symmetric gaits.
  • Curriculum Learning with Virtual Assistance: To overcome the difficulty of learning complex locomotion from scratch, they employ a curriculum learning strategy. This involves a "virtual assistant" that provides modulated physical assistance (balancing and propelling forces) to the character. This assistance is gradually reduced as the character's skill improves, effectively guiding it through increasingly challenging stages of learning. Two curriculum scheduling approaches are proposed:
  • Learner-centered: The curriculum adapts based on the agent's current performance, finding the next lesson that is challenging but still achievable.
  • Environment-centered: The curriculum follows a predefined path, exposing the agent to a range of lessons within each learning iteration to improve data efficiency.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated environments for training and evaluating locomotion controllers on various character models. The "datasets" are effectively the generated simulation trajectories and the resulting character behaviors.

Key Insights and Ideas

  • Symmetry in Actions, Not Just States: The insight that penalizing asymmetric actions is more effective and generalizable for learning symmetric gaits than penalizing state symmetry is crucial. This allows for more natural adaptation to different poses and situations.
  • Curriculum Learning as a "Training Wheel": The analogy of exoskeletons or training wheels for learning locomotion is powerful. The virtual assistant provides a scaffold that allows the agent to learn fundamental balance and propulsion without immediate failure, gradually removing the support as proficiency grows.
  • Emergence of Speed-Appropriate Gaits: A significant finding is that different gaits naturally emerge for different target velocities without explicit programming for each gait type. This suggests that the learned reward function and learning process are sufficient to discover these variations.
  • Minimalist Approach Yields Naturalness: The success of learning without motion capture or morphology-specific knowledge highlights the power of DRL when guided by well-chosen principles like symmetry and energy efficiency.

Key Results

  • The proposed method successfully generated symmetric and low-energy locomotion for a variety of character morphologies, including a simplified biped, a full humanoid, a quadruped, and a hexapod.
  • Learned gaits were observed to be speed-appropriate, with characters adopting walking gaits at lower speeds and running gaits (including flight phases) at higher speeds.
  • The environment-centered curriculum learning approach demonstrated superior data efficiency compared to the learner-centered approach.
  • Quantitative results showed significantly lower actuation magnitudes (energy consumption) and higher symmetry indices (e.g., SI values as low as 0.0026 for simplified biped walk) compared to baseline methods.
  • The method could also learn asymmetric gaits when required, such as when a humanoid character carried a heavy object.

Limitations and Future Work

  • Motion Quality vs. Human Animation: The authors acknowledge that while their learned motions are more natural than previous DRL attempts, they may not yet match the quality of motions derived from real-world motion capture or expert animation.
  • Generalization to Other Locomotion Types: The current work focuses on terrestrial locomotion. Future work could explore applying the curriculum learning to other forms of locomotion like swimming, flying, or soft-body movement.
  • Incorporating Motion Capture/Refinement: The authors suggest that their learned policies could serve as excellent starting points for further refinement using motion capture data or additional reward terms, potentially leading to even higher fidelity motions.
  • Sensitivity Analysis: A more comprehensive analysis of the impact of kinematic and dynamic properties, as well as torque ranges, on the resulting motion style is suggested.
  • Virtual Assistant Design: While the proposed virtual assistant is effective, further exploration into alternative designs and assistance strategies could yield further improvements.

Summary updated

Abstract

Learning locomotion skills is a challenging problem. To generate realistic and smooth locomotion, existing methods use motion capture, finite state machines or morphology-specific knowledge to guide the motion generation algorithms. Deep reinforcement learning (DRL) is a promising approach for the automatic creation of locomotion control. Indeed, a standard benchmark for DRL is to automatically create a running controller for a biped character from a simple reward function. Although several different DRL algorithms can successfully create a running controller, the resulting motions usually look nothing like a real runner. This paper takes a minimalist learning approach to the locomotion problem, without the use of motion examples, finite state machines, or morphology-specific knowledge. We introduce two modifications to the DRL approach that, when used together, produce locomotion behaviors that are symmetric, low-energy, and much closer to that of a real person. First, we introduce a new term to the loss function (not the reward function) that encourages symmetric actions. Second, we introduce a new curriculum learning method that provides modulated physical assistance to help the character with left/right balance and forward movement. The algorithm automatically computes appropriate assistance to the character and gradually relaxes this assistance, so that eventually the character learns to move entirely without help. Because our method does not make use of motion capture data, it can be applied to a variety of character morphologies. We demonstrate locomotion controllers for the lower half of a biped, a full humanoid, a quadruped, and a hexapod. Our results show that learned policies are able to produce symmetric, low-energy gaits. In addition, speed-appropriate gait patterns emerge without any guidance from motion examples or contact planning.
BibTeX
@article{yu2018learning,
  author = {Wenhao Yu and Greg Turk and C. Karen Liu},
  title = {{Learning Symmetry and Low-energy Locomotion}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  url = {https://openreview.net/forum?id=tENXD3IEDeG},
}
Figure