Probabilistic Modeling for Diverse Human Motion Synthesis

27 concepts24 papers1 proposal

When a human is asked to "walk forward," they might step off with their left foot, swing their arms widely, or tilt their torso slightly. There is no single correct way to execute an action; instead, a single high-level command maps to an infinite space of valid physical trajectories. This characteristic makes human movement fundamentally one-to-many.

Traditional deep learning approaches treat motion synthesis as a deterministic regression problem, optimizing models to predict exact joint coordinates. However, forcing a network to map a single input to a single deterministic output creates a severe mathematical conflict. When faced with multiple valid paths in the training data, a regression model minimizes its loss by averaging all possible outcomes. This "regression to the mean" causes synthesized characters to quickly lose momentum, dampening their movements until they collapse into a static, robotic, or frozen average pose.

To capture the natural, expressive variability of human movement, researchers have shifted from predicting specific coordinates to modeling the underlying probability distribution of motion. By treating synthesis as a stochastic sampling process, modern frameworks ensure that animations remain dynamic, diverse, and lifelike.


Mapping Distributions Instead of Points

The earliest solutions to the averaging problem introduced probabilistic latent spaces and density estimators. Instead of forcing the network to output a single next pose, these models predict the parameters of a continuous probability distribution from which a pose can be sampled.

Deterministic Regression:
Input Context ───> [ Neural Network ] ───> Single Averaged Pose (Static/Robotic)

Probabilistic Synthesis:
Input Context ───> [ Neural Network ] ───> Probability Distribution P(Pose) ───> Sampled Pose (Dynamic/Diverse)

Yujun Cai addressed this by reframing various synthesis tasks under a unified Conditional Variational Auto-Encoder (CVAE) framework in A Unified 3D Human Motion Synthesis Model via Conditional Variational Auto-Encoder∗, demonstrating that sampling from a conditional latent distribution successfully prevents characters from converging to a robotic mean. Similarly, chuan guo et al. designed a temporal VAE architecture in Generating Diverse and Natural 3D Human Motions from Text that uses "motion snippet codes" to generate multiple distinct, valid 3D animations from a single text prompt.

Other approaches model the temporal transitions explicitly as density functions. For instance, Benedikte Wallace et al. combined Recurrent Neural Networks with Mixture Density Networks in Towards Movement Generation with Audio Features to map complex audio features to dance movements, proving that non-deterministic density modeling is essential for creative improvisation. In A causal convolutional neural network for multi-subject motion modeling and generation, Shuaiying Hou et al. utilized causal convolutions to map latent features to a Gaussian probability density function rather than a single point estimate, allowing the system to sample varied, non-repetitive movements across different body proportions.

To prevent animations from freezing during long-term generation, Hongyu Tao developed a two-part transformer network in A Two-part Transformer Network for Controllable Motion Synthesis that predicts a probability distribution for future frames, ensuring agile and heterogeneous movements like cartwheels and dances remain fluid over time.

To capture even more complex, real-world behavior distributions, researchers integrated normalizing flows, which transform simple prior distributions into highly complex topologies through a series of invertible mappings. Andreas Blattmann et al. leveraged this in Behavior-Driven Synthesis of Human Dynamics to disentangle a motion's high-level behavior from its starting posture, enabling smooth transitions and behavioral re-enactment. Similarly, Wenjie Yin et al. developed Graph-based Normalizing Flow for Human Motion Generation and Reconstruction, combining normalizing flows with Graph Convolutional Networks to treat the skeleton as a structured graph, which drastically improves the stability and diversity of reconstructed motion capture data.


Discretization and Tokenized Motion

An alternative way to bypass the continuous averaging effect is to treat motion similarly to natural language. By discretizing continuous joint trajectories into a finite vocabulary of "motion tokens," models can use autoregressive classifiers to predict the probability of the next token, transforming motion synthesis into a discrete sampling problem.

Continuous Trajectory ───> [ Vector Quantization ] ───> Discrete Tokens: [Word A] [Word B] [Word C]
                                                                              │
Autoregressive Sampling <─── Predict Probability Distribution P(Next Token) <─┘

Saeed Ghorbani et al. pioneered this concept in Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model by representing motion sequences as high-level "Motion Words" within a probabilistic framework, allowing weak control signals like gender or action type to generate diverse, long-term animations.

This tokenization strategy forms the core of bidirectional text-motion frameworks. In TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts, Chuan Guo et al. discretized continuous motion into discrete tokens within an autoregressive framework. By modeling motion as a probability distribution over these tokens, their system avoids the deterministic limitations of regression models, generating multiple distinct motion sequences from a single text prompt.

This concept is equally powerful in speech-driven animation; the authors of Co-Speech Gesture Synthesis using Discrete Gesture Token Learning demonstrated that sampling from a learned probability distribution over discrete gesture states creates highly human-like, non-repetitive hand and body movements that align naturally with spoken audio.


The Diffusion Revolution

While VAEs and normalizing flows offer significant improvements over deterministic regression, they are prone to structural limitations. VAEs frequently suffer from posterior collapse, a failure mode where the latent space ignores the conditioning signals and reverts to a generic output. Generative Adversarial Networks (GANs) offer an alternative, but they are notoriously unstable and prone to mode collapse, capturing only a few safe motion variations.

To overcome these limitations, the field has widely adopted Denoising Diffusion Probabilistic Models (DDPMs). Diffusion models do not attempt to map an input directly to a motion sequence in a single step. Instead, they define a generative process that starts from pure Gaussian noise and iteratively removes noise over hundreds of small steps, conditioned on an input signal. Because the synthesis process originates from random noise, every single generation path is inherently stochastic, allowing the model to naturally cover the entire distribution of valid motions without collapsing.

Forward Process (Training):
Clean Motion ───> Add Noise (Step 1) ───> Add Noise (Step 2) ───> Pure Gaussian Noise

Reverse Process (Generation):
Pure Gaussian Noise + Condition ───> Denoise ───> Denoise ───> High-Fidelity Diverse Motion

Mingyuan Zhang et al. introduced this paradigm with MotionDiffuse in MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model, demonstrating that iterative denoising enables fine-grained, probabilistic synthesis with control over individual body parts and arbitrary sequence lengths. Concurrently, Sungjoon Choi et al. developed FLAME (FLAME: Free-form Language-based Motion Synthesis & Editing), introducing a "diffuse-then-denoise" strategy that enables precise text-based editing and motion in-betweening within a single framework, yielding significantly higher output diversity than traditional VAEs.

This superior stability and diversity have made diffusion models the standard across a variety of highly ambiguous, multi-modal synthesis tasks:

The shift from deterministic regression to conditional generation has even influenced low-level vision tasks. Ao Luo et al. reformulated optical flow estimation as a conditional generation task in FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models. By adopting a "noise-to-flow" strategy, their model captures the underlying probabilistic distribution of pixel displacements, demonstrating robust generalization against motion blur and atmospheric distortions where traditional regression models fail.

By framing motion as a landscape of probabilistic possibilities rather than a single deterministic path, these diverse architectures ensure that synthesized characters retain the inherent randomness, fluidity, and variety of real human behavior.

Key Foundational Work

Towards Movement Generation with Audio Features (2020) — contributed 1 concept elements to this direction.
Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model (2020) — contributed 1 concept elements to this direction.
Predicting Head Pose from Speech with a Conditional Variational Autoencoder (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_motion_synthesis_b1415b8b,
  title = {stochastic motion synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-motion-synthesis/rdr_b1415b8be94d4e96af8573ff338d671c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_motion_synthesis_b1415b8b_v1,
  title = {stochastic motion synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-motion-synthesis/rdr_b1415b8be94d4e96af8573ff338d671c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake