Mitigating Information Loss in Discrete Motion Tokenization

36 concepts33 papers1 proposal

The recent explosion of generative AI has driven a powerful trend in motion modeling: treating physical movement like language. By quantizing continuous 3D poses or robotic trajectories into a sequence of discrete tokens from a predefined codebook, researchers can cast complex motion generation and comprehension tasks as standard sequence-to-sequence problems (Walsh et al.). This tokenization strategy unlocks the massive scaling power of autoregressive transformers and Large Language Models (LLMs).

However, this convenience comes at a steep physical cost. Human and robotic movements are inherently fluid, continuous, and bound by complex spatial-temporal dynamics. Forcing these trajectories into a finite, discrete vocabulary introduces a fundamental representation gap—a "discretization error ceiling" that downstream models cannot overcome (Hong et al.).

The Geometry of Quantization Error

To understand why motion tokenization introduces artifacts, we must look at how vector quantization operates geometrically. A standard tokenizer maps a continuous trajectory segment to the single nearest centroid in a learned codebook.

As noted by HosseiniNejad et al., if a specific physical variation does not align perfectly with a cluster's precise average, the model is forced to retrieve an imprecise approximation. This limitation traces back to early insights by Ghosh et al., who demonstrated that discrete state models fail to recover accurate trajectories when complex physical gestures deviate from predefined centroids.

Continuous Trajectory:  ───╳────────╳────────╳─── (Fluid, exact positions)
                             │        │        │
Quantization Mapping:        ▼        ▼        ▼
Discrete Centroids:     [Centroid A] [Centroid B] [Centroid C]
                             │        │        │
Decoded Output:         ───■────────■────────■─── (Approximated, jittery)

This approximation acts as a lossy compression filter that smooths over critical kinematic details. Intricate finger movements, subtle joint rotations, and rapid velocity changes are often completely discarded during the discretization process because the codebook lacks the resolution to represent them (Walsh et al.; Wang et al.).

How Quantization Artifacts Manifest

When these discrete token sequences are decoded back into raw physical coordinates, the lost information manifests as distinct visual and physical artifacts.

  • Kinematic Inconsistencies: Chuan Guo et al. observed that tokenization errors frequently break physical constraints, resulting in visual blurriness or severe "foot sliding" in generated 3D meshes.
  • Jitter and Robotic Motion: Because a discrete vocabulary cannot natively represent smooth transitions between states, the resulting animations often appear robotic, unnatural, or plagued by high-frequency jitter (Chang et al.; Zhu et al.).
  • Trajectory Drift and Precision Loss: In robotics, the stakes are even higher. Song et al. found that discrete action tokens lead to accumulated resolution errors and trajectory drift during complex manipulation tasks. Furthermore, Kienle et al. showed that discrete tokenization completely fails to capture high-frequency physical events, such as the sudden force spikes that occur during contact-rich collisions.

Strategies for High-Fidelity Motion

To break through the performance ceiling imposed by quantization, researchers are actively developing two competing paradigms: refining the discrete codebook or abandoning tokenization entirely.

Multi-Scale and Residual Quantization

Rather than discarding discrete codebooks, one approach is to make them hierarchical so they can capture what a single-layer codebook misses. For instance, Liu et al. introduced a framework that quantizes common motion patterns while using a separate network to learn "rhythmic residuals" to recover subtle dynamics.

Similarly, the authors of MoMask (Guo et al.) and SnapMoGen (Guo et al.) utilize hierarchical Residual Vector Quantization (RVQ) and multi-scale tokenization. By encoding motion across multiple levels of coarse-to-fine detail, these architectures minimize precision loss and significantly reduce synthesis artifacts.

Native Continuous Modeling

A more radical shift involves treating motion as a native, continuous modality rather than forcing it into a language-like structure. Zhu et al. bypassed tokenization entirely by implementing a dual-stream architecture with a continuous diffusion bridge, preserving fine-grained kinematics and accelerating model convergence.

In the realm of multimodal LLMs, Li et al. developed a framework that preserves motion in its native continuous form, aligning it with text via a specialized cross-attention module rather than tokenizing it. For robotic control, Lin et al. leveraged continuous latent spaces and flow-based generative models to ensure that high-level intent and low-level execution remain mathematically consistent and fluid.

Key Foundational Work

Estimation of articulatory gesture patterns from speech acoustics (2009) — contributed 2 concept elements to this direction.
Recognizing Actions in 3D Using Action-Snippets and Activated Simplices (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_motion_quantization_artifacts_afeb76c7,
  title = {discrete motion quantization artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-motion-quantization-artifacts/rdr_afeb76c774f54013bbe9d86f7c7ffe6d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discrete_motion_quantization_artifacts_afeb76c7_v1,
  title = {discrete motion quantization artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-motion-quantization-artifacts/rdr_afeb76c774f54013bbe9d86f7c7ffe6d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake