Compositional Human Motion Synthesis

36 concepts33 papers1 proposal

Imagine asking a text-to-motion model to generate an animation for a simple prompt: "a person walks forward." Modern generative models handle this effortlessly. But what happens when the prompt becomes a narrative? "A person hears a noise, turns around, walks forward while waving, and then sits down."

Faced with such multi-stage or simultaneous instructions, standard text-to-motion models suffer from action collapse or a compositional complexity ceiling. Instead of executing each action sequentially or layering them smoothly, the model often treats the prompt as a unordered "bag of words." This causes it to blend distinct behaviors into a static, physically impossible mess, or omit actions entirely. As noted by Chuan Guo et al. in their work introducing the Generating Diverse and Natural 3D Human Motions from Text benchmark, models struggle to clearly delineate individual sub-actions when forced to process a complex description monolithically.

To overcome this limitation, recent research shifts from treating motion generation as a single end-to-end mapping to a compositional framework. This approach breaks down complex actions across two primary dimensions: time (sequencing actions chronologically) and space (layering actions across different body parts).

Temporal Composition: Decomposing the Timeline

When a human executes a sequence of actions, there is a clear temporal and causal logic. Standard text-to-motion architectures pass the entire text prompt through a global encoder (like CLIP), which flattens this chronological progression. The model loses track of what happens first, second, and last.

To preserve the sequence, researchers use a divide-and-conquer strategy. Yijun Qian et al. proposed Breaking The Limits of Text-conditioned 3D Motion Synthesis with Elaborative Descriptions, which factorizes long narratives into discrete "atomic actions." By explicitly segmenting the text, the model can generate motion snippets for each sub-action sequentially rather than attempting to synthesize the entire sequence at once.

Building on this, Jaeyoung Seon introduced Event-T2M: Event-level Conditioning for Complex Text-to-Motion Synthesis, which uses event-level cross-attention to map specific time windows of the generation process to corresponding semantic "events" in the text. This prevents the model from bleeding future actions into the present, ensuring that the character finishes turning around before they start walking.

Similarly, Runqi Ouyang et al. developed Motion-R1: Enhancing Motion Generation with Decomposed Chain-of-Thought and RL Binding, which leverages a decomposed chain-of-thought mechanism to explicitly reason about the temporal transitions and causal logic before generating the physical joint rotations.

Spatial Composition: Disentangling Body Parts

Temporal sequencing handles then, but how do we handle while? Generating simultaneous actions—such as "waving while walking"—presents a spatial composition challenge. If a model tries to learn "waving while walking" as a single monolithic category, it suffers from severe data scarcity, as capturing every possible combination of human movements in a dataset is impossible.

The solution is to decouple the human body into independent kinematic zones. In SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation, Nikos Athanasiou et al. use large language models to parse text prompts and assign specific actions to specific body parts (e.g., upper body waves, lower body walks). By synthesizing these part-specific motions independently and then fusing them, the model can generate novel, unseen combinations of simultaneous behaviors.

This part-based factorization is also mirrored in frameworks like MoRAG - Multi-Fusion Retrieval Augmented Generation for Human Motion, which retrieves separate motion components for the hands, torso, and legs, fusing them into a unified, physically plausible animation.

Latent Space Composition and Logical Operators

Beyond explicit structural splitting, composition can be achieved mathematically within the latent spaces of generative models. Instead of interpolating linearly between action classes—which Zhenyi Wang et al. demonstrate in Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions is restricted to simple combinations—modern approaches treat composition as an energy optimization problem.

In EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space, Hehe Fan et al. frame motion generation using energy-based diffusion. By defining separate "semantic-aware" energy functions for different actions, they can compose complex motions using strict logical operators like AND (conjunction) and NOT (negation). For instance, to generate "walking but NOT waving," the model minimizes the energy associated with walking while maximizing the energy associated with waving, allowing for precise algebraic control over the generated behavior directly within the diffusion model's latent space.

Stitching it Together: Continuity and Evaluation

Decomposing actions into temporal events or spatial parts solves the composition problem, but it introduces a new challenge: continuity. How do you ensure that the transition between "running" and "sitting" doesn't look like a jarring, unnatural jump?

Germán Barquero addresses this in Seamless Human Motion Composition with Blended Positional Encodings by introducing a mechanism that blends relative and absolute positional encodings within a diffusion model. This allows the model to smoothly extrapolate periodic motions and stitch sequential prompts together without requiring artificial post-processing or inverse kinematics smoothing.

To validate these compositional models, traditional metrics that evaluate single actions are insufficient. Kent Fujiwara et al. introduced the Chronologically Accurate Retrieval for Temporal Grounding of Motion-Language Models (CAR) benchmark, which specifically tests a model's awareness of event order by using shuffled descriptions as hard negatives during training. Similarly, Xinpeng Liu proposed Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases, establishing an intermediate representation called Kinematic Phrases to evaluate how well models handle repetitive, sequential, or simultaneous movements.

Key Foundational Work

Predictive Coding-based Deep Dynamic Neural Network for Visuomotor Learning (2017) — contributed 1 concept elements to this direction.
Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_motion_synthesis_5991659a,
  title = {compositional motion synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-motion-synthesis/rdr_5991659adcc84c09957f1dc1e9d6c151},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_motion_synthesis_5991659a_v1,
  title = {compositional motion synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-motion-synthesis/rdr_5991659adcc84c09957f1dc1e9d6c151/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake