Large Language Models (LLMs) have mastered the nuances of human text, but physical movement—the "body language" of 3D skeletal joints and facial expressions—has traditionally been treated as a continuous signal, making it difficult to integrate into the discrete world of Transformers. A rapidly growing body of research is bridging this gap by treating 3D human motion as a "foreign language." By discretizing continuous physical movements into a finite vocabulary of tokens, researchers are enabling LLMs to reason about, plan, and generate human actions with the same autoregressive power they use for prose.
The Motion Vocabulary: From Joint Angles to Tokens
The core challenge in teaching an LLM to "speak" motion is the high-dimensional, continuous nature of 3D data. A single second of movement involves dozens of joint rotations and translations. To solve this, researchers like Thomas Lucas in PoseGPT and Biao Jiang in MotionGPT employ a Vector Quantized Variational Autoencoder (VQ-VAE).
The intuition is simple: just as a language has a finite set of words, human movement can be broken down into a finite set of "motion primitives." The VQ-VAE encoder compresses a sequence of 3D poses into a latent space, which is then mapped to the nearest entry in a learned "codebook." This turns a complex dance or a simple wave into a sequence of discrete integers—tokens—that an LLM can process alongside text.
To capture finer details without an exponentially large codebook, some frameworks use more sophisticated quantization. For instance, Jen-Hao Rick Chang and colleagues at Apple utilize Residual Vector Quantization to represent facial motions in a coarse-to-fine manner (Probabilistic Speech-Driven 3D Facial Motion Synthesis), while PoseGPT uses Product Quantization to partition feature vectors, allowing for high-fidelity reconstruction of complex hand and face gestures.
Motion-Language Pre-training
Once motion is tokenized, the LLM treats it as just another modality. In MotionGPT: Human Motion as a Foreign Language, the model is trained on a unified vocabulary of both text and motion tokens. This allows the model to perform "translation" tasks:
- Text-to-Motion: "Translate" the sentence "a person walks in a circle" into a sequence of motion tokens.
- Motion Captioning: "Translate" a sequence of motion tokens into a descriptive sentence.
- Motion Prediction: Treat a partial sequence of motion tokens as a prompt and "autocomplete" the rest of the movement.
This paradigm shift allows for extreme efficiency. Yaqi Zhang and Lu Chen demonstrated in MotionGPT: Finetuned LLMs are General-Purpose Motion Generators that by fine-tuning only 0.4% of a pre-trained LLM's parameters (like LLaMA), the model can inherit the reasoning capabilities of the base LLM to generate realistic animations from multimodal prompts.
From Generation to Interactive Planning
The real power of the "motion-as-language" approach lies in high-level reasoning. Traditional motion models are "one-shot"—you give a prompt, and they produce a clip. By contrast, frameworks like AvatarGPT by Zixiang Zhou and Baoyuan Wang treat motion as part of a closed-loop system.
Because the LLM understands the "semantics" of the motion tokens, it can handle complex, multi-step instructions. If a user asks an avatar to "prepare the office for a meeting," AvatarGPT doesn't just generate a single walk; it plans a sequence of actions—walking to a chair, pulling it out, picking up a laptop—and generates the corresponding motion tokens for each step. Similarly, Motion-Agent by Qi Wu enables conversational motion editing, where a user can refine a movement through multi-turn dialogue, much like chatting with an AI assistant to edit a paragraph of text.
Expanding the Embodiment
The discrete tokenization strategy is proving robust across different types of "bodies":
- Social Interaction: SOLAMI uses tokenized motion to power autonomous characters in VR, allowing them to respond to human speech with synchronized body language.
- Sign Language: Weichao Zhao developed BEST, which uses "coupling tokenization" to capture the spatial correlation between hand gestures and body posture for sign language recognition.
- Robotics: Humanoid-LLA by Kaiyang Ji maps diverse motion data into a unified vocabulary, allowing a humanoid robot to execute whole-body commands from free-form language.
- Cross-Species Retargeting: MoReFlow by Wontaek Kim uses discrete tokens to transfer motion between characters with different skeletal structures (e.g., human to quadruped), focusing on high-level spatiotemporal patterns rather than raw joint angles.
By moving away from continuous coordinate regression and toward discrete sequence modeling, this research direction is turning the "black box" of 3D animation into a legible, programmable language that LLMs can finally understand.