Generating realistic 3D human motion—whether it is a person walking, a hand grasping a mug, or a speaker gesturing during a lecture—is a fundamental challenge in computer vision. While we have nearly infinite text and images to train models like GPT-4 or Stable Diffusion, high-quality 3D motion capture (MoCap) data is notoriously scarce. Capturing 3D movement requires expensive multi-camera rigs, wearable sensors, and professional actors, making it impossible to "scrape" the physical world at the scale of the internet.
To overcome this bottleneck, researchers are moving away from pure supervised learning. Instead, they are developing methods to "borrow" knowledge from text, audio, and images to synthesize 3D motion without requiring massive, hand-labeled datasets.
Motion as a Discrete Language
One of the most powerful shifts in this field is treating 3D motion not as a series of raw coordinates, but as a "language." Just as a sentence is a sequence of discrete words, a complex movement can be broken down into a sequence of discrete "motion tokens."
Changan Chen and colleagues at Stanford demonstrate this in The Language of Motion. By tokenizing different body parts separately, they allow a single Multimodal Large Language Model (LLM) to process motion, speech, and text in the same mathematical space. This works because it reframes motion synthesis as a "next-token prediction" problem—the same mechanism that allows LLMs to write poetry. When motion is discretized, models can learn the "grammar" of how a human body moves, even when paired data is limited.
Similarly, Teo Guichoux and the team behind Gelina use an interleaved token strategy. By training on massive amounts of speech data first and then fine-tuning on smaller multimodal sets, the model learns to "fill in the blanks" for gestures, effectively using the abundance of audio data to support the scarcity of 3D motion data.
Wordless Training via Latent Alignment
If we don't have enough pairs of "text + motion," can we train a model using text it has never seen paired with motion? This is the core idea behind "wordless training."
In Being Comes from Not-Being, Jianlong Chang leverages the CLIP latent space. CLIP is already trained on billions of image-text pairs; it "knows" what the concept of "running" looks like in a visual sense. By mapping 3D poses directly into this pre-existing CLIP space, the model can generate motion from arbitrary text prompts without ever seeing a single text-motion pair during training. The model essentially "hallucinates" the motion by following the semantic breadcrumbs left by CLIP.
Synthesizing the Missing Ground Truth
When real-world data doesn't exist, some researchers simply build it. This "fake it to make it" approach uses specialized AI chains to create massive synthetic datasets.
For example, Shivam Mehta and the creators of MAGI addressed the shortage of synchronized speech-and-gesture data by generating 38 hours of artificial training data. By chaining together text-to-speech and motion-synthesis models, they created a "bootstrap" dataset that allowed their final model to achieve higher naturalness than models trained on limited human data.
This concept extends to specialized domains like "air-writing." Songbin Xu developed the Air-Writing Translater to solve the lack of paired sensor-to-trajectory data. Instead of trying to record thousands of people writing in the air with expensive trackers, they used unsupervised domain adaptation to translate "invisible" motion sensor readings into spatial coordinates, effectively bridging the gap between what a sensor "feels" and what a hand "draws."
Decoupling Semantics from Physics
A recurring theme in recent work is the separation of what an action is from how it is physically executed. By decoupling these two, models can generalize to new scenarios more easily.
- Human-Object Interaction: Sirui Xu introduced InterDreamer, which uses an LLM to plan the semantics of an interaction (e.g., "pick up the chair") and a physics-aware world model to handle the actual 3D movement. This zero-shot approach avoids the need for massive datasets of people interacting with every possible object.
- Structural Augmentation: Jiahang Zhang and the SGAR framework decompose full-body movements into part-level descriptions. Instead of learning "a person jumping," the model learns what the legs do versus what the arms do. This granular understanding allows the model to "mix and match" body parts, effectively augmenting the limited training data into a much larger variety of possible motions.
By treating motion as a language, leveraging pre-trained multimodal spaces like CLIP, and synthesizing artificial training data, researchers are finally breaking the data bottleneck that has long constrained 3D human animation.