villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen, Jiang Bian

Abstract

Visual-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent work has begun to explore the incorporation of latent actions, an abstract representation of visual change between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. Together, these contributions enable villa-X to achieve superior performance across simulated environments including SIMPLER and LIBERO, as well as on two real-world robot setups including gripper and dexterous hand manipulation. We believe the ViLLA paradigm holds significant promise, and that our villa-X provides a strong foundation for future research.

Introduction

Vision-Language-Action (VLA) models serve as the foundational control systems for modern general-purpose robots, translating visual inputs and linguistic instructions into precise motor commands. To scale these models effectively, researchers have increasingly turned to latent actions—compact, abstract representations of motion between video frames. These tokens allow models to leverage vast amounts of "action-free" data, such as human-centric videos, where explicit motor signals are absent.

However, a significant challenge remains: most existing latent action models are grounded purely in pixel-level changes. This approach often fails to capture critical control signals—such as subtle wrist rotations or minute gripper adjustments—that have a minimal visual footprint but are essential for task success. Without physical grounding, these "un-grounded" latents struggle to transfer knowledge from human demonstrations to real-world robotic hardware.

Methodology

The villa-X framework addresses these limitations through two primary innovations: a physically grounded latent action model and a hierarchical policy architecture.

Physically Grounded Latent Action Model (LAM)

Standard latent action models typically employ an Inverse Dynamics Model (IDM) to extract latents and a visual Forward Dynamics Model (FDM) for pixel reconstruction. villa-X enhances this by introducing a proprioceptive FDM (proprio-FDM). This auxiliary module forces latent tokens to predict future robot states (joint positions) and actions, anchoring the abstract representations in physical reality. To accommodate diverse robotic platforms, an "embodiment context" vector ($c_e$) is utilized to account for variations in control frequencies and morphologies.

Comparison of the existing latent action model (a) and the proposed physically grounded villa-X model (b).

The ACTor Module (ACT)

Rather than using latents solely for pre-training, villa-X implements a hierarchical policy via the ACTor module, consisting of two specialized components:

  1. ACT-latent: A mid-level planner that generates a sequence of future latent action tokens based on visual and linguistic context.
  2. ACT-robot: A low-level controller that produces actual robot action "chunks," conditioned on the latent plan provided by ACT-latent.
The villa-X ACTor architecture, showing the hierarchical flow from VLM to ACT-latent and finally to ACT-robot.

Both components are trained using a conditional flow matching framework (a type of diffusion). A stochastic masking strategy is employed during training to ensure the robot-action branch remains robust and avoids learning simplistic shortcuts.

Experiments and Results

villa-X was pre-trained on a massive dataset comprising 1.6 million robot trajectories (223.5M frames) and 3.6 million human video clips from sources like Open X-Embodiment and Ego4D.

Simulation Performance

In the SIMPLER benchmark, villa-X demonstrated significant improvements over established baselines. On the Google Robot platform, it achieved a 77.7% average success rate, substantially outperforming RT-1-X (49.4%) and OpenVLA (32.7%).

Real-World and Dexterous Manipulation

Real-world evaluations on a Realman arm showed 100% success on "Pick out" and "Unstack" tasks, outperforming GR00T. Furthermore, villa-X exhibited strong performance in dexterous manipulation using a 12-DoF XHand, proving the scalability of the grounded latent approach to complex control spaces.

Zero-Shot Generalization

The model demonstrated remarkable zero-shot capabilities, generating valid latent plans for unseen robotic embodiments and interpreting open-vocabulary symbolic icons (e.g., "touch the corn" via a card icon) without specific training on these symbols.

Conclusion

villa-X advances the state-of-the-art in VLA modeling by bridging the gap between visual perception and physical dynamics. By grounding latent actions in proprioception and employing a hierarchical planning structure, the framework achieves superior generalization and robustness. Future work will explore deeper planning horizons, self-correction mechanisms via foundation model critics, and alternative grounding cues such as human hand pose estimation.

Summary updated

Abstract

Visual-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent work has begun to explore the incorporation of latent actions, an abstract representation of visual change between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. Together, these contributions enable villa-X to achieve superior performance across simulated environments including SIMPLER and LIBERO, as well as on two real-world robot setups including gripper and dexterous hand manipulation. We believe the ViLLA paradigm holds significant promise, and that our villa-X provides a strong foundation for future research.
BibTeX
@inproceedings{chen2026villa,
  author = {Xiaoyu Chen and Hangxing Wei and Pushi Zhang and Chuheng Zhang and Kaixin Wang and Yanjiang Guo and Rushuai Yang and Yucen Wang and Xinquan Xiao and Li Zhao and Jianyu Chen and Jiang Bian},
  title = {{villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=y5CaJb17Fn},
}
Figure