For years, computer vision treated human movement as a series of raw 3D coordinates—a continuous stream of numbers representing joints in space. While precise, this representation is fundamentally different from how we describe actions. We don't think of a wave as a set of XYZ trajectories; we think of it as a discrete semantic unit.
A growing body of research suggests that the key to mastering human dynamics is to treat motion not as a signal to be regressed, but as a "foreign language" to be translated. By discretizing continuous movement into a vocabulary of "motion tokens," researchers are now leveraging the reasoning power of Large Language Models (LLMs) to generate, understand, and edit human actions with unprecedented flexibility.
The Motion Tokenizer
The bridge between physical movement and linguistic reasoning is the motion tokenizer. Just as a text tokenizer breaks sentences into sub-word units, researchers like Kenneth Li and Xiao Sun have proposed breaking complex dynamics into a sequence of fundamental "motion words" derived from a learned lexicon of recurring patterns.
This is typically achieved using a Vector Quantized-Variational AutoEncoder (VQ-VAE). As Evonne Ng and colleagues demonstrated in Learning to Listen, a model can learn a discrete codebook where each entry represents a short, human-like motion segment. When a continuous skeleton sequence is passed through this bottleneck, it is mapped to the nearest codes in the library.
The result is a "string of motion." A backflip becomes a sequence like [Token_42, Token_109, Token_7]. This discretization acts as a semantic filter; as Weichao Zhao notes in the BEST framework, predicting discrete tokens is often more effective than regressing raw coordinates because it forces the model to ignore low-level jitter and focus on the underlying "grammar" of the gesture.
Motion as a Linguistic Modality
Once motion is tokenized, it becomes compatible with the Transformer architectures that power modern LLMs. In MotionGPT, Biao Jiang and his team explicitly treat 3D motion as a specialized dialect. By expanding an LLM’s vocabulary to include these motion tokens, the model can perform "next-token prediction" across both text and movement.
This unification allows the model to inherit the world knowledge and reasoning capabilities of the LLM. For example:
- Semantic Alignment: The model learns that the text token "jump" frequently co-occurs with specific motion tokens, enabling high-fidelity text-to-motion generation.
- Contextual Reasoning: In GesTran, Chenghao Xu uses LLMs to translate speech into gestures, leveraging the model's understanding of intent to generate movements that are contextually appropriate and synchronized with audio.
- Task Planning: Zixiang Zhou introduced AvatarGPT, which uses this "body language" to bridge high-level goals (e.g., "prepare the office") with low-level physical execution, treating the entire process as a single, continuous linguistic sequence.
Zero-Shot Editing and Interaction
The power of the language metaphor is most evident in motion editing. If motion is a sentence, editing it should be as simple as changing a word.
The CoMo framework by Yiming Huang exploits this by allowing users to perform "zero-shot" edits through natural language. Because the motion is represented as interpretable "pose codes," the LLM can identify which tokens in a sequence correspond to a specific body part (like a "bent knee") and swap them for tokens that match a new instruction (like "straighten leg") while keeping the rest of the "sentence" intact.
This linguistic approach also extends to social and environmental interactions. Jianping Jiang and the SOLAMI team have developed Vision-Language-Action (VLA) models where an autonomous character’s speech and body language are predicted within the same unified backbone. Similarly, HSI-GPT treats the interaction between a human and a 3D scene as a next-token prediction task, allowing the model to "read" the geometry of a room and "write" the appropriate motion tokens to navigate it.
Why It Works: The Power of Structure
The shift toward tokenization works because it solves the "alignment problem" between modalities. Continuous signals are messy and high-dimensional; language is structured and symbolic. By forcing motion into a symbolic format, we allow models to apply the same predictive logic to a dance that they apply to a paragraph.
As Yuan Wang demonstrates in MotionGPT-2, this doesn't just make generation easier—it makes the models "smarter" about movement. When an LLM "speaks" motion, it isn't just mimicking coordinates; it is navigating a structured space of human intention, resulting in movements that are more coherent, editable, and semantically meaningful.