Abstract
Visual-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent work has begun to explore the incorporation of latent actions, an abstract representation of visual change between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. Together, these contributions enable villa-X to achieve superior performance across simulated environments including SIMPLER and LIBERO, as well as on two real-world robot setups including gripper and dexterous hand manipulation. We believe the ViLLA paradigm holds significant promise, and that our villa-X provides a strong foundation for future research.
Introduction
Vision-Language-Action (VLA) models serve as the foundational control systems for modern general-purpose robots, translating visual inputs and linguistic instructions into precise motor commands. To scale these models effectively, researchers have increasingly turned to latent actions—compact, abstract representations of motion between video frames. These tokens allow models to leverage vast amounts of "action-free" data, such as human-centric videos, where explicit motor signals are absent.
However, a significant challenge remains: most existing latent action models are grounded purely in pixel-level changes. This approach often fails to capture critical control signals—such as subtle wrist rotations or minute gripper adjustments—that have a minimal visual footprint but are essential for task success. Without physical grounding, these "un-grounded" latents struggle to transfer knowledge from human demonstrations to real-world robotic hardware.
Methodology
The villa-X framework addresses these limitations through two primary innovations: a physically grounded latent action model and a hierarchical policy architecture.
Physically Grounded Latent Action Model (LAM)
Standard latent action models typically employ an Inverse Dynamics Model (IDM) to extract latents and a visual Forward Dynamics Model (FDM) for pixel reconstruction. villa-X enhances this by introducing a proprioceptive FDM (proprio-FDM). This auxiliary module forces latent tokens to predict future robot states (joint positions) and actions, anchoring the abstract representations in physical reality. To accommodate diverse robotic platforms, an "embodiment context" vector ($c_e$) is utilized to account for variations in control frequencies and morphologies.
The ACTor Module (ACT)
Rather than using latents solely for pre-training, villa-X implements a hierarchical policy via the ACTor module, consisting of two specialized components:
- ACT-latent: A mid-level planner that generates a sequence of future latent action tokens based on visual and linguistic context.
- ACT-robot: A low-level controller that produces actual robot action "chunks," conditioned on the latent plan provided by ACT-latent.
Both components are trained using a conditional flow matching framework (a type of diffusion). A stochastic masking strategy is employed during training to ensure the robot-action branch remains robust and avoids learning simplistic shortcuts.
Experiments and Results
villa-X was pre-trained on a massive dataset comprising 1.6 million robot trajectories (223.5M frames) and 3.6 million human video clips from sources like Open X-Embodiment and Ego4D.
Simulation Performance
In the SIMPLER benchmark, villa-X demonstrated significant improvements over established baselines. On the Google Robot platform, it achieved a 77.7% average success rate, substantially outperforming RT-1-X (49.4%) and OpenVLA (32.7%).
Real-World and Dexterous Manipulation
Real-world evaluations on a Realman arm showed 100% success on "Pick out" and "Unstack" tasks, outperforming GR00T. Furthermore, villa-X exhibited strong performance in dexterous manipulation using a 12-DoF XHand, proving the scalability of the grounded latent approach to complex control spaces.
Zero-Shot Generalization
The model demonstrated remarkable zero-shot capabilities, generating valid latent plans for unseen robotic embodiments and interpreting open-vocabulary symbolic icons (e.g., "touch the corn" via a card icon) without specific training on these symbols.
Conclusion
villa-X advances the state-of-the-art in VLA modeling by bridging the gap between visual perception and physical dynamics. By grounding latent actions in proprioception and employing a hierarchical planning structure, the framework achieves superior generalization and robustness. Future work will explore deeper planning horizons, self-correction mechanisms via foundation model critics, and alternative grounding cues such as human hand pose estimation.
Summary updated