Spatio-Temporal Masking for Editable Motion Generation

This direction is merged into another direction: foot sliding artifacts. Its last version stays readable and citable here.

Version 1 of 114 papers21 conceptsfrozen 2026-10-04

Traditional 3D animation is a painstaking process of balancing physical realism with creative intent. While generative models have simplified the creation of motion from scratch, they often act as "black boxes" that offer little control over the final output. If an animator likes a generated dance but wants to change just the arm movements, or needs to bridge two distinct poses, they previously had to restart or manually fix the frames.

A new paradigm in motion synthesis treats animation not as a fixed sequence, but as a signal that can be partially "masked" and reconstructed. By leveraging the iterative nature of diffusion models, researchers have found that we can lock specific joints or time segments and let the model "fill in the blanks" (inpainting). This approach turns generative models into flexible, interactive tools for precise motion editing.

Motion as an Inpainting Task

The core intuition behind this research direction is to frame motion editing as a spatio-temporal inpainting problem. In image generation, inpainting involves masking a portion of a picture and asking the model to generate content that blends seamlessly with the unmasked surroundings. In 3D motion, this mask can be applied across two dimensions:

  1. Temporal Masking: Fixing specific frames (like the beginning and end of a sequence) and asking the model to generate the "in-between" motion.
  2. Spatial Masking: Fixing specific body parts (like the legs during a walk) and asking the model to generate new movements for others (like the arms waving).

As Jonathan Tseng, Rodrigo Castellon, and Karen Liu demonstrate in EDGE, this is achieved through masked denoising. During the diffusion process, the model iteratively removes noise to create a clean motion. By overwriting the masked regions with "ground truth" data at every step, the model is forced to synthesize the remaining parts so they are both physically plausible and perfectly synchronized with the fixed constraints.

Zero-Shot Control Without Retraining

One of the most powerful aspects of this mechanism is that it requires no task-specific training. A model trained simply to generate motion can perform complex editing tasks "out of the box."

In SinMDM, Sigal Raab, Guy Tevet, and colleagues show that even a model trained on a single motion sequence can perform style transfer and motion expansion. By injecting structural information or spatial masks during the inference-time denoising process, the model adapts the learned motion to new constraints.

Similarly, FLAME, developed by Sungjoon Choi and his team, uses a "diffuse-then-denoise" strategy. This allows for text-based editing where a user can describe a change (e.g., "waving while walking"), and the model reconstructs the motion under the guidance of a binary mask, preserving the original gait while modifying the upper body.

Unified Synthesis and Refinement

This masking approach effectively unifies different animation tasks—synthesis, editing, and prediction—into a single framework. Jianxin Ma and the team at Alibaba’s DAMO Academy proposed MoFusion to act as a pretrained backbone for these diverse tasks. By treating joints as fixed constraints, the model ensures that any generated motion remains consistent with the user's requirements.

The flexibility of this method extends beyond full-body human motion:

  • Facial Animation: Inkyu Park and Jaewoong Cho introduced SAiD, which uses diffusion inpainting to modify specific facial features or temporal segments in speech-driven animation.
  • Co-Speech Gestures: Haiyang Liu and colleagues developed EMAGE, which allows users to provide "partial motion hints." The model then fills in the missing body movements, transforming the AI from a simple generator into a collaborative tool for animators.

Granular Modeling for Better Control

To improve the precision of these edits, recent work has moved toward more granular representations of the body. Weihao Yuan and the authors of MoGenTS argue that quantizing the entire body into a single token loses detail. Instead, they model each joint individually in a 2D spatial-temporal "token map." This allows the masked modeling framework to lock specific joints with much higher fidelity, enabling subtle edits—like changing a hand gesture—without affecting the rest of the skeletal structure.

By treating motion as a multi-dimensional signal subject to spatio-temporal constraints, these models move away from "all-or-nothing" generation. They provide a mathematical bridge between the creative freedom of generative AI and the precise control required by professional animators.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatio_temporal_masking_for_editable_mot_9e6d6e9c,
  title = {Spatio-Temporal Masking for Editable Motion Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-masking-for-editable-motion-generation/rdr_9e6d6e9c2f9e4ba0a88b824d97286f89},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatio_temporal_masking_for_editable_mot_9e6d6e9c_v1,
  title = {Spatio-Temporal Masking for Editable Motion Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-masking-for-editable-motion-generation/rdr_9e6d6e9c2f9e4ba0a88b824d97286f89/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}