Learning to Get Up

Tianxin Tao, Matthew Wilson, Ruiyu Gou, Michiel van de Panne

TL;DR

This paper introduces a novel three-stage reinforcement learning framework for generating natural and diverse human "get-up" motions from arbitrary fallen poses, without relying on motion capture data. The method progressively learns to discover efficient get-up strategies, adapt them to weaker character models, and finally execute them at slower, more human-like speeds, even allowing for pauses in stable intermediate poses.

What Problem Does This Paper Address?

Simulating realistic human behavior in virtual environments is crucial for applications ranging from gaming to robotics. A fundamental, yet challenging, aspect of this is enabling characters to recover from falls and return to a standing posture. Existing approaches often fall into two camps: either they rely on motion capture data, which limits the diversity and adaptability of the learned motions, or they use reinforcement learning (RL) but tend to produce overly dynamic, erratic, and unnatural movements. This paper aims to bridge this gap by developing an RL-based method that generates human-like, diverse, and controllable get-up behaviors from scratch, without requiring pre-recorded motion data.

Methods

The core of the proposed solution is a staged learning pipeline:

  1. Discover (Strong Character): The process begins by training an initial "get-up" policy on a character with high torque limits (a "strong" character). This allows for broader exploration of the state-action space, making it easier to discover viable "solution modes" for getting up and avoiding local optima that might lead to unnatural motions.
  2. Weaker Curriculum: The learned policy is then progressively refined by gradually reducing the character's torque limits. This "strong-to-weak curriculum" encourages the policy to learn more energy-efficient and natural movements, mimicking the observed tendency for humans to use less forceful strategies when possible.
  3. Slower Imitation: Finally, a separate imitation-learning policy is trained to reproduce the motions learned in the "weaker" stage, but at significantly slower speeds (up to 5x slower). This is achieved by retiming the trajectories generated by the "weaker" policy and training a new controller to follow these slower paths. This stage also incorporates a "future pose conditioning" mechanism, allowing the policy to anticipate and maintain intermediate poses, enabling paused get-ups.

The authors use Soft Actor-Critic (SAC) as their RL algorithm, known for its sample efficiency and ability to balance exploration and exploitation.

Datasets

The paper does not explicitly mention or use external datasets for training or evaluation. Instead, it relies on a physics-based simulator (Mujoco) to generate its own training environments and trajectories. The "datasets" are effectively the simulated experiences and the generated trajectories from the different learning stages.

Key Insights and Ideas

  • Strength for Exploration, Weakness for Naturalness: The paper highlights a crucial insight: strong actuation (high torque limits) is beneficial for initial exploration and discovering how to get up, while weaker actuation is key to learning natural and human-like movements.
  • Curriculum Learning is Essential: The staged approach, particularly the strong-to-weak curriculum, is shown to be vital for overcoming the challenges of learning slow, low-effort motions and avoiding common RL pitfalls like local minima.
  • Future Pose Conditioning for Control: Conditioning the "slower" policy on future poses allows for more nuanced control, enabling not just slower motions but also the ability to pause in statically stable intermediate states.
  • Diversity from Randomization: The method naturally discovers a diverse range of get-up strategies, both across different training runs and even within a single controller when presented with different initial states (e.g., prone vs. supine).

Key Results

  • The proposed framework successfully generates natural and diverse human get-up motions from arbitrary fallen poses without motion capture data.
  • The learned controllers can execute get-up motions at various speeds, controlled by a retiming coefficient.
  • The generated motions exhibit significant static stability, allowing them to be paused at certain points.
  • The method demonstrates robustness by successfully generating get-up motions for modified humanoid characters, such as those with limbs in casts or missing limbs.
  • Ablation studies confirm the necessity of the strong-to-weak curriculum and the slow imitation stage for achieving high-quality, natural motions.

Limitations and Future Work

  • Two-Stage Trajectory Generation: The current pipeline requires a separate simulation run with the "weaker" policy to generate reference trajectories for the "slower" policy. The authors suggest that a single policy conditioned on the current state and desired speed k could be learned via distillation, eliminating this intermediate step.
  • Lack of User Control: The current system discovers diverse strategies but doesn't offer direct user control over the choice of strategy. Future work could involve learning a set of controllers and enabling users to select their preferred strategy.
  • Generalization to Diverse Scenarios: While the method works for constrained scenarios like casts, generalizing to more complex environments (e.g., chairs, bathtubs, uneven terrain, or wearing skates) remains an open challenge.
  • Simplified Humanoid Model: The current humanoid model is simplified (e.g., missing fingers/toes), which could limit the realism of the motions. More detailed modeling is suggested for improved realism.

Summary updated

Abstract

Getting up from an arbitrary fallen state is a basic human skill. Existing methods for learning this skill often generate highly dynamic and erratic get-up motions, which do not resemble human get-up strategies, or are based on tracking recorded human get-up motions. In this paper, we present a staged approach using reinforcement learning, without recourse to motion capture data. The method first takes advantage of a strong character model, which facilitates the discovery of solution modes. A second stage then learns to adapt the control policy to work with progressively weaker versions of the character. Finally, a third stage learns control policies that can reproduce the weaker get-up motions at much slower speeds. We show that across multiple runs, the method can discover a diverse variety of get-up strategies, and execute them at a variety of speeds. The results usually produce policies that use a final stand-up strategy that is common to the recovery motions seen from all initial states. However, we also find policies for which different strategies are seen for prone and supine initial fallen states. The learned get-up control strategies often have significant static stability, i.e., they can be paused at a variety of points during the get-up motion. We further test our method on novel constrained scenarios, such as having a leg and an arm in a cast.
BibTeX
@inproceedings{tao2022learning,
  author = {Tianxin Tao and Matthew Wilson and Ruiyu Gou and Michiel van de Panne},
  title = {{Learning to Get Up}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2022},
  url = {https://openreview.net/forum?id=nc8GMAGC76},
}
Figure