Flexible Motion In-betweening with Diffusion Models

Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, Michiel van de Panne

Abstract

Generating smooth, realistic transitions between character keyframes—a process known as in-betweening—is a fundamental yet labor-intensive task in animation. This paper presents CondMDI (Conditional Motion Diffusion In-betweening), a diffusion-based framework designed to automate this process with unprecedented flexibility. By utilizing a masked diffusion training strategy, CondMDI learns to synthesize high-quality human motion from arbitrary spatial and temporal constraints, including sparse keyframes, partial body trajectories, and natural language prompts.

Introduction

In character animation, "in-betweening" bridges the gap between user-defined keyframes to create fluid movement. While deep learning has introduced automated solutions, many existing models suffer from rigidity, requiring fixed keyframe intervals or struggling to explore the diverse range of possible movements between two points. Furthermore, most models cannot handle "partial" constraints, such as defining only a hand's path without specifying the rest of the skeleton. CondMDI addresses these challenges by treating in-betweening as a conditional generation task within a unified diffusion framework.

Methodology

CondMDI is built upon the principle that a model trained to reconstruct missing data can effectively serve as a flexible in-betweener. The core components of the framework include:

Masked Diffusion Training

Unlike standard diffusion models that rely on inference-time guidance to satisfy constraints, CondMDI is trained from scratch to handle missing information. During training, the model is presented with motion sequences where frames and joints are randomly masked. This "masked diffusion" approach teaches the model to reconstruct full, coherent motion based on whatever unmasked "anchors" remain, making it inherently agnostic to the type or timing of input constraints.

Global Root Representation

To avoid the error accumulation common in relative root position datasets, the authors adopt a global coordinate system. This allows the model to anchor characters precisely in space and time, which is critical for long-range in-betweening where a character must reach a specific destination.

Architecture and Conditioning

The system utilizes a 1D UNet backbone (derived from the GMD framework) and incorporates semantic control via a pre-trained CLIP text encoder. This allows users to guide the style and intent of the motion (e.g., "a person walks and then sits") while the diffusion process handles the physical realization of the movement.

Experimental Setup and Datasets

The model was trained and evaluated using the HumanML3D dataset, which consists of 14,646 human motion sequences. Each sequence is paired with multiple text descriptions, providing a rich foundation for multi-modal conditioning. The dataset covers a wide variety of actions derived from AMASS and HumanAct12, ensuring the model's robustness across different movement types.

Key Insights

  • Training vs. Guidance: A significant finding is that training a model specifically for masked reconstruction outperforms inference-time "reconstruction guidance." While guidance often results in physical implausibility or "jumps" at keyframes, CondMDI produces smoother, more natural transitions.
  • Generative Diversity: As a diffusion model, CondMDI does not simply interpolate linearly. It can "hallucinate" multiple plausible ways to reach a target—such as choosing between a hop or a step—while remaining faithful to the provided text prompt.
  • Sparse Joint Control: The model demonstrates high efficacy in inferring full-body motion from minimal data, such as "VR joints" (head and wrists). It can accurately predict the movement of the torso and legs based solely on these sparse inputs.

Results

CondMDI demonstrates superior performance across several metrics:

  • Precision: In scenarios with 20 randomly placed keyframes, the model achieved a Keyframe Error of 0.0754, outperforming general-purpose diffusion models.
  • Quality: The Fréchet Inception Distance (FID) remains competitive with state-of-the-art unconditioned models, indicating that the constraints do not degrade the realism of the motion.
  • Efficiency: The model is significantly faster than complex alternatives like OmniControl, generating motions in approximately 54 seconds on mid-range hardware (RTX 2070).
  • Versatility: Qualitative tests show the model successfully handling complex, dynamic movements like yoga sequences and martial arts from extremely sparse inputs.

Limitations and Future Work

Despite its flexibility, CondMDI faces challenges common to deep learning motion models, such as "footskate" (minor sliding of the feet) and high-frequency jitter in dynamic sequences. Future work may involve integrating physics-based loss functions to improve ground contact and exploring more efficient data representations to allow for even finer control, such as individual finger manipulation. Additionally, grounding the training process in specific animator workflows could further enhance its utility in professional production environments.

Summary updated

BibTeX
@inproceedings{cohan2024flexible,
  author = {Setareh Cohan and Guy Tevet and Daniele Reda and Xue Bin Peng and Michiel van de Panne},
  title = {{Flexible Motion In-betweening with Diffusion Models}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657414},
}
Figure