MotionVLA: Vision-Language-Action Model for Humanoid Motion

Nonghai Zhang, Siyu Zhai, Yanjun Li, Zeyu Zhang, Zhihan Yin, Yandong Guo, Boxin Shi, Hao TangarXiv 2026 · arXiv

TL;DR

MotionVLA introduces a novel dual-stream frequency tokenizer (DSFT) that decouples low-frequency pose semantics from high-frequency physical dynamics in humanoid motion generation. By modeling these distinct streams sequentially within an autoregressive vision-language-action framework, the model significantly reduces common synthesis artifacts like foot sliding and temporal drift.

What Problem Does This Paper Address?

Generating realistic humanoid motion from text descriptions and scene images requires capturing both the overarching pose structure (semantics) and fine-grained physical movements (dynamics). Current state-of-the-art autoregressive models typically compress all motion data into a single, shared token codebook.

The authors reveal a fundamental flaw in this unified approach: human motion is spectrally heterogeneous. Joint positions are highly low-frequency; just five Discrete Cosine Transform (DCT) coefficients can capture 93% of their energy. Conversely, joint velocities are high-frequency, with those same five coefficients capturing only 37% of their energy.

When forced into a single codebook, the low-frequency position statistics dominate the vocabulary. The model treats high-frequency velocity signals as noise, leading to physical errors that accumulate over time—manifesting as joint instability, temporal drift, and "foot sliding."

Figure 3: Frequency-domain clustering of motion dimensions. (a) Per-dimension low-frequency ratio on HumanML3D. (b) Corresponding histogram on HumanML3D. (c/d) Corresponding plots on ViMoGen. Both datasets exhibit a consistent bimodal separation between low-frequency Base dimensions and high-frequency Phys dimensions.

Methods

To resolve this statistical mismatch, the authors propose MotionVLA, which consists of two primary innovations:

  1. Dual-Stream Frequency Tokenizer (DSFT): Instead of a single codebook, DSFT splits motion into two separate streams: a Base stream (capturing joint positions and rotations) and a Phys stream (capturing joint velocities). It applies DCT independently to both streams but uses different truncation lengths ($Kb = 5$ for Base, $Kp = 25$ for Phys) to respect their unique frequency profiles. Each stream is then compressed independently using Byte-Pair Encoding (BPE).
  2. Hierarchical Autoregressive Modeling: MotionVLA utilizes a Qwen3.5 language model backbone. It arranges the tokens in a unified sequence where all Base tokens are predicted before the Phys tokens. This specific ordering allows the causal attention mechanism to leverage the complete semantic pose context when generating the high-frequency physical dynamics. During inference, a phase-aware logit mask enforces this strict generation order.
Figure 2: Overview of MotionVLA. (a) DSFT performs dual-stream frequency tokenization by decomposing motion into Base and Phys components and converting them into discrete tokens. (b) During training, MotionVLA learns to autoregressively predict the unified motion token sequence under text and scene-image conditioning, supervised by DSFT tokens derived from ground-truth motion. (c) At inference time, the model generates Base and Phys tokens conditioned on multimodal inputs, which are then decoded and recombined to reconstruct the final motion sequence.

Datasets

The model was trained and evaluated on the following datasets:

  • ViMoGen-228K: A large-scale multimodal dataset containing 228,000 motion sequences from optical motion capture, in-the-wild videos, and synthetic generation.
  • MBench: The associated evaluation benchmark for ViMoGen, evaluating fine-grained physical quality across eight dimensions.
  • HumanML3D: A standard text-to-motion benchmark consisting of 14,616 motion clips, used here to test text-only generalization.

Key Insights and Ideas

  • The Spectral Mismatch: Treating positions and velocities identically during tokenization is the root cause of physical artifacts in autoregressive motion models. Decoupling them orthogonally before quantization preserves low-level motion details without sacrificing high-level structure.
  • Causal Ordering Matters: Generating physical tokens after semantic tokens mirrors human biomechanics—physical dynamics are naturally conditioned on intended pose trajectories.
  • Pointwise Error vs. Token Quality: Interestingly, the authors found that lower pointwise reconstruction errors (like root mean square error) do not automatically mean a better tokenizer. DSFT had slightly higher pointwise errors than a single-stream baseline but achieved significantly better downstream generation quality and lower Frechet Distance scores.

Key Results

Despite using a lightweight 2B parameter backbone, MotionVLA demonstrated substantial improvements over existing methods:

  • On MBench: It achieved the highest scores in Motion-Condition Consistency (improving from 0.53 to 0.55) and significantly reduced Foot Sliding (down to 0.0049).
  • On HumanML3D: It reduced the Diversity gap to real data by over 50% and achieved the highest multimodal diversity (MModality) among all compared generative methods.
  • Human Evaluation: In a blinded preference study conducted with domain experts, MotionVLA's generated motions were preferred in 64.0% of pairwise comparisons against the ViMoGen baseline.
Figure 1: Given a text description and a scene video as input, MotionVLA generates motions that closely track the ground truth (GT) across the full sequence (frames at 30%, 60%, and 90% shown). ViMoGen [13], which relies on a single-stream tokenizer, exhibits temporal drift and joint instability that accumulate over time (highlighted in white circles).

Limitations and Future Work

The authors note that their study was limited to a 2B parameter backbone, meaning the scaling behavior of this dual-stream approach remains unexamined. Additionally, the framework relies on a fixed stream partition, fixed truncation lengths, and a rigid Base-to-Phys generation order that might not be optimal for every type of human movement. Future work will focus on testing larger model scales, exploring broader datasets, and developing more adaptive tokenization schemes.

Summary updated

Abstract

Generating realistic humanoid motion from scene images and text involves both low-frequency pose semantics and high-frequency physical dynamics. However, many existing methods tokenize motion with a single shared codebook, forcing heterogeneous motion signals into the same quantization space. Our frequency-domain analysis of human motion data reveals a clear mismatch between single-codebook quantization and motion statistics: five DCT coefficients capture 93% of joint-position energy but only 37% of joint-velocity energy, which can bias quantization toward pose statistics and under-represent high-frequency velocity components. A second challenge lies in adapting a standard autoregressive model to effectively model high-frequency physical signals in motion sequences. Therefore, we propose DSFT, a dual-stream frequency tokenizer that separates motion into Base and physical streams and compresses them independently with DCT truncation and BPE. Furthermore, we present MotionVLA, a Qwen3.5-based model that arranges Base and physical tokens in a unified sequence, where Phys tokens are predicted after Base tokens. Experiments on HumanML3D and MBench show that, despite using a lightweight 2B backbone, MotionVLA reduces the Diversity gap to real data by over 50% on HumanML3D and improves Motion-Condition Consistency by 3.8% on MBench, supporting frequency-aware dual-stream decoupling as an effective formulation for autoregressive motion generation. Code: https://github.com/AIGeeksGroup/MotionVLA. Website: https://aigeeksgroup.github.io/MotionVLA.
BibTeX
@misc{zhang2026motionvla,
  author = {Nonghai Zhang and Siyu Zhai and Yanjun Li and Zeyu Zhang and Zhihan Yin and Yandong Guo and Boxin Shi and Hao Tang},
  title = {{MotionVLA: Vision-Language-Action Model for Humanoid Motion}},
  year = {2026},
  eprint = {2606.15142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.15142},
}

Loading related content...

Figure