When a human is asked to "walk forward," they might step off with their left foot, swing their arms widely, or tilt their torso slightly. There is no single correct way to execute an action; instead, a single high-level command maps to an infinite space of valid physical trajectories. This characteristic makes human movement fundamentally one-to-many.
Traditional deep learning approaches treat motion synthesis as a deterministic regression problem, optimizing models to predict exact joint coordinates. However, forcing a network to map a single input to a single deterministic output creates a severe mathematical conflict. When faced with multiple valid paths in the training data, a regression model minimizes its loss by averaging all possible outcomes. This "regression to the mean" causes synthesized characters to quickly lose momentum, dampening their movements until they collapse into a static, robotic, or frozen average pose.
To capture the natural, expressive variability of human movement, researchers have shifted from predicting specific coordinates to modeling the underlying probability distribution of motion. By treating synthesis as a stochastic sampling process, modern frameworks ensure that animations remain dynamic, diverse, and lifelike.
Mapping Distributions Instead of Points
The earliest solutions to the averaging problem introduced probabilistic latent spaces and density estimators. Instead of forcing the network to output a single next pose, these models predict the parameters of a continuous probability distribution from which a pose can be sampled.
Deterministic Regression:
Input Context ───> [ Neural Network ] ───> Single Averaged Pose (Static/Robotic)
Probabilistic Synthesis:
Input Context ───> [ Neural Network ] ───> Probability Distribution P(Pose) ───> Sampled Pose (Dynamic/Diverse)
Yujun Cai addressed this by reframing various synthesis tasks under a unified Conditional Variational Auto-Encoder (CVAE) framework in A Unified 3D Human Motion Synthesis Model via Conditional Variational Auto-Encoder∗, demonstrating that sampling from a conditional latent distribution successfully prevents characters from converging to a robotic mean. Similarly, chuan guo et al. designed a temporal VAE architecture in Generating Diverse and Natural 3D Human Motions from Text that uses "motion snippet codes" to generate multiple distinct, valid 3D animations from a single text prompt.
Other approaches model the temporal transitions explicitly as density functions. For instance, Benedikte Wallace et al. combined Recurrent Neural Networks with Mixture Density Networks in Towards Movement Generation with Audio Features to map complex audio features to dance movements, proving that non-deterministic density modeling is essential for creative improvisation. In A causal convolutional neural network for multi-subject motion modeling and generation, Shuaiying Hou et al. utilized causal convolutions to map latent features to a Gaussian probability density function rather than a single point estimate, allowing the system to sample varied, non-repetitive movements across different body proportions.
To prevent animations from freezing during long-term generation, Hongyu Tao developed a two-part transformer network in A Two-part Transformer Network for Controllable Motion Synthesis that predicts a probability distribution for future frames, ensuring agile and heterogeneous movements like cartwheels and dances remain fluid over time.
To capture even more complex, real-world behavior distributions, researchers integrated normalizing flows, which transform simple prior distributions into highly complex topologies through a series of invertible mappings. Andreas Blattmann et al. leveraged this in Behavior-Driven Synthesis of Human Dynamics to disentangle a motion's high-level behavior from its starting posture, enabling smooth transitions and behavioral re-enactment. Similarly, Wenjie Yin et al. developed Graph-based Normalizing Flow for Human Motion Generation and Reconstruction, combining normalizing flows with Graph Convolutional Networks to treat the skeleton as a structured graph, which drastically improves the stability and diversity of reconstructed motion capture data.
Discretization and Tokenized Motion
An alternative way to bypass the continuous averaging effect is to treat motion similarly to natural language. By discretizing continuous joint trajectories into a finite vocabulary of "motion tokens," models can use autoregressive classifiers to predict the probability of the next token, transforming motion synthesis into a discrete sampling problem.
Continuous Trajectory ───> [ Vector Quantization ] ───> Discrete Tokens: [Word A] [Word B] [Word C]
│
Autoregressive Sampling <─── Predict Probability Distribution P(Next Token) <─┘
Saeed Ghorbani et al. pioneered this concept in Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model by representing motion sequences as high-level "Motion Words" within a probabilistic framework, allowing weak control signals like gender or action type to generate diverse, long-term animations.
This tokenization strategy forms the core of bidirectional text-motion frameworks. In TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts, Chuan Guo et al. discretized continuous motion into discrete tokens within an autoregressive framework. By modeling motion as a probability distribution over these tokens, their system avoids the deterministic limitations of regression models, generating multiple distinct motion sequences from a single text prompt.
This concept is equally powerful in speech-driven animation; the authors of Co-Speech Gesture Synthesis using Discrete Gesture Token Learning demonstrated that sampling from a learned probability distribution over discrete gesture states creates highly human-like, non-repetitive hand and body movements that align naturally with spoken audio.
The Diffusion Revolution
While VAEs and normalizing flows offer significant improvements over deterministic regression, they are prone to structural limitations. VAEs frequently suffer from posterior collapse, a failure mode where the latent space ignores the conditioning signals and reverts to a generic output. Generative Adversarial Networks (GANs) offer an alternative, but they are notoriously unstable and prone to mode collapse, capturing only a few safe motion variations.
To overcome these limitations, the field has widely adopted Denoising Diffusion Probabilistic Models (DDPMs). Diffusion models do not attempt to map an input directly to a motion sequence in a single step. Instead, they define a generative process that starts from pure Gaussian noise and iteratively removes noise over hundreds of small steps, conditioned on an input signal. Because the synthesis process originates from random noise, every single generation path is inherently stochastic, allowing the model to naturally cover the entire distribution of valid motions without collapsing.
Forward Process (Training):
Clean Motion ───> Add Noise (Step 1) ───> Add Noise (Step 2) ───> Pure Gaussian Noise
Reverse Process (Generation):
Pure Gaussian Noise + Condition ───> Denoise ───> Denoise ───> High-Fidelity Diverse Motion
Mingyuan Zhang et al. introduced this paradigm with MotionDiffuse in MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model, demonstrating that iterative denoising enables fine-grained, probabilistic synthesis with control over individual body parts and arbitrary sequence lengths. Concurrently, Sungjoon Choi et al. developed FLAME (FLAME: Free-form Language-based Motion Synthesis & Editing), introducing a "diffuse-then-denoise" strategy that enables precise text-based editing and motion in-betweening within a single framework, yielding significantly higher output diversity than traditional VAEs.
This superior stability and diversity have made diffusion models the standard across a variety of highly ambiguous, multi-modal synthesis tasks:
- Speech and Music Gestures: In The DiffuseStyleGesture+ entry to the GENEA Challenge 2023, Sicheng Yang et al. showed that initiating synthesis from Gaussian noise effectively mitigates the repetitive, robotic behaviors common in traditional gesture generation. For complex rhythmic movements, Zhuoran Zhao et al. developed Diffusion-Conductor (Taming Diffusion Models for Music-Driven Conducting Motion Generation), bypassing unstable GAN architectures to capture the broad distribution of orchestral conducting. To capture expressive musical performances, Qijun Gan et al. utilized diffusion baselines in PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance to generate continuous, realistic hand fingerings directly from piano audio.
- Multi-Agent Interactions: Baptiste Chopin et al. extended diffusion to multi-person dynamics in Bipartite Graph Diffusion Model for Human Interaction Generation, using a bipartite graph module within a transformer to model the spatial-temporal cross-talk between two interacting skeletons without collapsing toward a mean pose.
- Sparse or Ambiguous Conditioning: Inferring full-body motion from limited sensors is highly ill-posed. Jiaman Li addressed this in Ego-Body Pose Estimation via Ego-Head Pose Estimation by using a conditional diffusion model to generate full-body motion from a single head trajectory, successfully mapping the one-to-many relationship between head movement and body pose. Similarly, Angela Castillo et al. developed BoDiffusion (BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis), demonstrating that diffusing sparse tracking data from the head and hands avoids posterior collapse, significantly reducing artifacts like "foot skating."
The shift from deterministic regression to conditional generation has even influenced low-level vision tasks. Ao Luo et al. reformulated optical flow estimation as a conditional generation task in FlowDiffuser: Advancing Optical Flow Estimation with Diffusion Models. By adopting a "noise-to-flow" strategy, their model captures the underlying probabilistic distribution of pixel displacements, demonstrating robust generalization against motion blur and atmospheric distortions where traditional regression models fail.
By framing motion as a landscape of probabilistic possibilities rather than a single deterministic path, these diverse architectures ensure that synthesized characters retain the inherent randomness, fluidity, and variety of real human behavior.
Key Foundational Work
Towards Movement Generation with Audio Features (2020) — contributed 1 concept elements to this direction.
Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model (2020) — contributed 1 concept elements to this direction.
Predicting Head Pose from Speech with a Conditional Variational Autoencoder (2017) — contributed 1 concept elements to this direction.