For years, the field of computer vision and graphics treated human and object motion through a highly fragmented lens. If you wanted to predict where a walking person would step next, you built a forecasting model. If you wanted to animate a character from a text prompt, you built a text-to-motion generator. If you wanted to transfer the dance moves of a human to a cartoon character or a toy dinosaur, you had to engineer custom structural templates or skeletons.
Each task, input modality, and object category required its own specialized architecture, loss function, and training pipeline. This fragmentation created a significant bottleneck: models could not share representations across related tasks, and they failed to generalize when presented with novel object shapes or unconstrained movements.
Recently, a powerful paradigm shift has emerged. Instead of building specialized "heads" for distinct tasks, researchers are unifying motion modeling by treating movement either as a universal sequence-completion problem or as an implicit, representation-agnostic field. This approach allows a single generalist architecture to seamlessly handle diverse tasks, modalities, and topologies.
Unifying Tasks via Masked Sequence Modeling
The core intuition behind modern unified motion models is that all temporal synthesis tasks—whether forecasting the future, filling in missing frames, or generating motion from scratch—can be framed as a single mathematical problem: reconstructing masked data.
Consider how human motion is structured. It is a continuous sequence of spatial coordinates or joint rotations over time. If we discretize or patchify this sequence, we can treat it exactly like text in a Large Language Model (LLM) or patches in a Vision Transformer.
By framing motion synthesis as a "masked sequence" problem, Yujun Cai et al. demonstrated that a single architecture could handle both motion prediction and motion completion without task-specific modifications. Under this framework:
- Motion Forecasting is simply masking the end of a sequence and asking the model to fill it in based on the historical context.
- Motion In-betweening is masking the middle of a sequence and forcing the model to find a physically plausible trajectory between a known start and end state.
- Unconditioned Generation is masking the entire sequence and letting the model sample from its learned distribution.
To make this sequence modeling highly effective, researchers often quantize continuous 3D movements into discrete tokens, effectively turning motion into a "foreign language." For instance, Thomas Lucas et al. utilized a discrete latent space to bridge the gap between unconditioned generation and context-aware prediction.
Building on this tokenization strategy, Lu Chen et al. introduced a prompt-based framework where motion tokens and text instructions coexist in a single multimodal sequence. This allows a user to change the model's behavior from a forecaster to a text-driven animator simply by altering the text prompt, bypassing the need for task-specific fine-tuning. Similarly, Esteve Valls Mascaro et al. showed that rethinking motion synthesis as a masked reconstruction problem on patchified skeletons allows a single model to generalize across various temporal and spatial masking scenarios fluidly.
Task Unification via Masking:
[ Frame 1 ] [ Frame 2 ] [ MASKED ] [ MASKED ] --> Motion Forecasting
[ Frame 1 ] [ MASKED ] [ MASKED ] [ Frame 4 ] --> Motion In-betweening
[ MASKED ] [ MASKED ] [ MASKED ] [ MASKED ] --> Unconditioned Generation
Breaking Modality and Granularity Silos
Human movement does not happen in a vacuum; it is driven by language, audio, and surrounding geometry. Traditional deep learning models were typically restricted to a single input modality—such as text-to-motion or music-to-dance—which limited their flexibility in real-world workflows where multiple control signals are required simultaneously.
To break these silos, generalist frameworks scale up both the dataset diversity and the architectural capacity. Zixiang Zhou et al. developed a unified driving engine capable of generating motion from both natural language and audio sequences within a single model, enabling seamless transitions between different driving modalities.
When scene geometry is introduced, the next-token prediction paradigm scales even further. For example, Ya-Li Li et al. unified 3D scene geometry, human motion, and natural language into a single LLM framework, allowing the system to fluidly transition between synthesizing actions within a new environment and generating textual descriptions of existing movements.
Furthermore, Bizhu Wu et al. demonstrated that a single architecture can bridge the gap between high-level comprehension (like temporal localization and captioning) and low-level execution (precise motion generation), proving that understanding motion and generating motion are two sides of the same conceptual coin.
Eliminating Category-Specific Structural Priors
Beyond unifying tasks, the field has faced a significant hurdle in generalizing motion across different topologies. Traditional motion transfer and pose transformation techniques relied heavily on category-specific geometric priors, such as human skeletal rigs, facial landmarks, or predefined 3D templates. If you wanted to transfer motion to an object that did not match the template—like an animal or an arbitrary consumer product—the system broke down.
The key to solving this problem is moving away from explicit structural anchors toward implicit or representation-agnostic motion spaces. Instead of tracking specific joints, modern frameworks learn how pixels, volumes, or latent features deform globally.
Xinpeng Liu tackled this by proposing a homogeneous dynamics space that acts as a representation-agnostic bridge, consolidating motor knowledge from diverse sensing technologies into a single reusable model. For image animation and pose transformation, Yukun Su et al. bypassed category-specific structural priors entirely by using dual-warping techniques on unpaired data, allowing a unified framework to transform the poses of humans, animals, and vehicles alike.
Another elegant solution involves extracting motion directly from the internal feature spaces of pre-trained generative models. Danah Yatim et al. leveraged the space-time attention maps of pre-trained text-to-video diffusion models to perform zero-shot motion transfer. Because these features capture how regions of an image shift over time regardless of the specific object category, they allow the structural topology of the subject to change completely (e.g., animating a dolphin using the motion of a dog) while perfectly preserving the underlying dynamics.
Similarly, Junyi Wu et al. used internal spatial-temporal priors to create controllable velocity fields for rigid motion transfer, ensuring that arbitrary objects can be animated realistically without requiring any predefined structural models.
By treating motion as a universal sequence to be completed or an implicit field to be deformed, the field is successfully moving past the era of fragmented, task-specific engineering, paving the way for true general-purpose motion intelligence.
Key Foundational Work
A Joint Model for 2D and 3D Pose Estimation from a Single Image (2013) — contributed 1 concept elements to this direction.
Lie-X: Depth Image Based Articulated Object Pose Estimation, Tracking, and Action Recognition on Lie Groups (2016) — contributed 1 concept elements to this direction.