villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen, Jiang Bian

TL;DR

villa-X introduces a novel Vision-Language-Latent-Action framework that bridges the gap between purely visual video pre-training and physical robot control. By grounding abstract "latent actions" with a proprioceptive Forward Dynamics Model and using a joint diffusion policy, villa-X achieves state-of-the-art manipulation performance and demonstrates zero-shot generalization across entirely unseen robot embodiments.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models are powerful, but training them requires massive amounts of data. To bypass the scarcity of real robot trajectories, researchers often use Latent Action Models (LAMs) to learn abstract motion representations from abundant, action-free human videos.

However, standard LAMs rely entirely on visual reconstruction. This creates a major flaw: critical control movements—like subtle gripper rotations or fine finger adjustments—cause very small pixel changes. Purely vision-based models tend to ignore these subtle motions, leading to "physically ungrounded" latent actions. When a VLA model tries to translate these ungrounded tokens into real-world robot commands, the performance drops significantly, especially when transferring knowledge across different robot types (heterogeneous embodiments).

Methods

To solve this, the authors propose villa-X, which improves both how latent actions are learned and how they are integrated into VLA policies.

1. Physically Grounded Latent Action Model (LAM)

Instead of predicting only the next video frame, villa-X introduces an auxiliary decoder called a proprioceptive Forward Dynamics Model (proprio-FDM). When robot data is available during pre-training, this module forces the latent actions ($z_t$) to predict future physical robot states ($q$) and low-level actions ($a$). To handle data from different robots without muddying the latent space, they condition the proprio-FDM on an embodiment context vector ($c_e$) containing dataset IDs and control frequencies.

Figure 1: (a) A standard Latent Action Model (LAM) learns a latent action zt primarily through visual reconstruction, predicting a future frame ˆot+K from the current frame ot and latent action zt. (b) Our proposed model enhances this by adding a proprio-FDM. This auxiliary module predicts future robot states ˆqt+1:t+K and actions ˆat:t+K−1 conditioned on an embodiment context ce, enabling the latent actions to be better grounded in physical dynamics.

2. Joint Diffusion Policy (ACTor Module)

To effectively use these high-quality latents, the authors design the ACTor Module (ACT). Rather than treating latent actions as a simple initialization step, ACT models the joint distribution of both latent plans and low-level robot actions using a conditional flow matching (diffusion) framework.

To prevent the robot-action branch from taking shortcuts and overly relying on the latent tokens, they implement a stochastic masking strategy: during training, attention from robot actions to latent actions is stochastically blocked or masked out 50% of the time.

Figure 2: Architecture of ACT: A hierarchical policy that predicts latent action plans and conditions robot action generation on them, incorporating embodiment context and attention masking.

Datasets

The model underwent large-scale pre-training on a massive data mixture:

  • Robot Data: 1.6 million trajectories (223.5 million frames) combining the Open X-Embodiment mixture and AgiBot datasets.
  • Human Video Data: 3.6 million video clips from datasets including Ego4D (EgoHOD subset), EgoPAT3D, EPIC-KITCHENS, HO-Cap, HOI4D, HoloAssist, RH20T, and Something-Something V2.
  • Evaluation Benchmarks: Evaluated in simulation via SIMPLER (Google Robot and WidowX variants) and LIBERO (4 task suites), alongside real-world deployments on a 6-DoF Realman arm (gripper) and a 7-DoF XArm equipped with a 12-DoF XHand (dexterous manipulator).

Key Insights and Ideas

  • Proprioception as a Visual Anchor: Forcing the latent space to solve a parallel physical prediction task acts as a regularizer. It ensures that the visual features captured by the model actually correspond to actionable physical dynamics.
  • True Cross-Embodiment Transfer: Because the latent action space is stripped of robot-specific quirks via the context vector ($c_e$), the latent planner becomes completely embodiment-agnostic. It can generate valid motion plans for robots it has never seen before during training.
  • Co-Dependence via Diffusion: Jointly diffusing the latent plan and the low-level execution path allows the model to continuously refine its physical actions based on its mid-level visual plan, leading to much smoother and more robust control.

Key Results

Simulation Benchmarks

  • SIMPLER: villa-X achieved the highest average success rates on both the Google Robot setup (77.7%) and the WidowX Robot setup (62.5%), outperforming strong baselines like OpenVLA, $\pi_0$, and visual-trace methods like Magma.
  • LIBERO: On the 4 task suites, villa-X achieved an average success rate of 90.1%, a significant jump over the next best baseline, $\pi_0$-FAST (85.5%). Ablating the latent action expert dropped performance to 81.9%, proving its necessity.

Real-World Robots

  • Realman Arm (Gripper): villa-X achieved perfect success rates (100%) on tasks like "Pick out" and "Unstack," outperforming GR00T by a wide margin.
  • XArm + XHand (Dexterous Hand): Since no dexterous hand data was used during pre-training, this setup tested true embodiment transfer. villa-X outperformed specialized baselines like GR-1 and GR00T, hitting an 84% success rate on seen pick-and-place tasks and 68% on unseen variations.

Limitations and Future Work

While the latent expert is highly effective at future planning, the authors note that its full generative potential is not yet fully exploited. Currently, it outputs a single deterministic plan path.

For future work, they propose integrating a VLM-based critic module. This would allow the latent expert to sample multiple diverse trajectory plans, while the critic rejects any plans that fail to align with the high-level language instructions before they are sent to the robot's physical controllers.

Summary updated

Abstract

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.
BibTeX
@inproceedings{chen2026villa,
  author = {Xiaoyu Chen and Hangxing Wei and Pushi Zhang and Chuheng Zhang and Kaixin Wang and Yanjiang Guo and Rushuai Yang and Yucen Wang and Xinquan Xiao and Li Zhao and Jianyu Chen and Jiang Bian},
  title = {{villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=y5CaJb17Fn},
}
Figure