TL;DR
villa-X introduces a novel Vision-Language-Latent-Action framework that bridges the gap between purely visual video pre-training and physical robot control. By grounding abstract "latent actions" with a proprioceptive Forward Dynamics Model and using a joint diffusion policy, villa-X achieves state-of-the-art manipulation performance and demonstrates zero-shot generalization across entirely unseen robot embodiments.
What Problem Does This Paper Address?
Vision-Language-Action (VLA) models are powerful, but training them requires massive amounts of data. To bypass the scarcity of real robot trajectories, researchers often use Latent Action Models (LAMs) to learn abstract motion representations from abundant, action-free human videos.
However, standard LAMs rely entirely on visual reconstruction. This creates a major flaw: critical control movements—like subtle gripper rotations or fine finger adjustments—cause very small pixel changes. Purely vision-based models tend to ignore these subtle motions, leading to "physically ungrounded" latent actions. When a VLA model tries to translate these ungrounded tokens into real-world robot commands, the performance drops significantly, especially when transferring knowledge across different robot types (heterogeneous embodiments).
Methods
To solve this, the authors propose villa-X, which improves both how latent actions are learned and how they are integrated into VLA policies.
1. Physically Grounded Latent Action Model (LAM)
Instead of predicting only the next video frame, villa-X introduces an auxiliary decoder called a proprioceptive Forward Dynamics Model (proprio-FDM). When robot data is available during pre-training, this module forces the latent actions ($z_t$) to predict future physical robot states ($q$) and low-level actions ($a$). To handle data from different robots without muddying the latent space, they condition the proprio-FDM on an embodiment context vector ($c_e$) containing dataset IDs and control frequencies.
2. Joint Diffusion Policy (ACTor Module)
To effectively use these high-quality latents, the authors design the ACTor Module (ACT). Rather than treating latent actions as a simple initialization step, ACT models the joint distribution of both latent plans and low-level robot actions using a conditional flow matching (diffusion) framework.
To prevent the robot-action branch from taking shortcuts and overly relying on the latent tokens, they implement a stochastic masking strategy: during training, attention from robot actions to latent actions is stochastically blocked or masked out 50% of the time.
Datasets
The model underwent large-scale pre-training on a massive data mixture:
- Robot Data: 1.6 million trajectories (223.5 million frames) combining the Open X-Embodiment mixture and AgiBot datasets.
- Human Video Data: 3.6 million video clips from datasets including Ego4D (EgoHOD subset), EgoPAT3D, EPIC-KITCHENS, HO-Cap, HOI4D, HoloAssist, RH20T, and Something-Something V2.
- Evaluation Benchmarks: Evaluated in simulation via SIMPLER (Google Robot and WidowX variants) and LIBERO (4 task suites), alongside real-world deployments on a 6-DoF Realman arm (gripper) and a 7-DoF XArm equipped with a 12-DoF XHand (dexterous manipulator).
Key Insights and Ideas
- Proprioception as a Visual Anchor: Forcing the latent space to solve a parallel physical prediction task acts as a regularizer. It ensures that the visual features captured by the model actually correspond to actionable physical dynamics.
- True Cross-Embodiment Transfer: Because the latent action space is stripped of robot-specific quirks via the context vector ($c_e$), the latent planner becomes completely embodiment-agnostic. It can generate valid motion plans for robots it has never seen before during training.
- Co-Dependence via Diffusion: Jointly diffusing the latent plan and the low-level execution path allows the model to continuously refine its physical actions based on its mid-level visual plan, leading to much smoother and more robust control.
Key Results
Simulation Benchmarks
- SIMPLER: villa-X achieved the highest average success rates on both the Google Robot setup (77.7%) and the WidowX Robot setup (62.5%), outperforming strong baselines like OpenVLA, $\pi_0$, and visual-trace methods like Magma.
- LIBERO: On the 4 task suites, villa-X achieved an average success rate of 90.1%, a significant jump over the next best baseline, $\pi_0$-FAST (85.5%). Ablating the latent action expert dropped performance to 81.9%, proving its necessity.
Real-World Robots
- Realman Arm (Gripper): villa-X achieved perfect success rates (100%) on tasks like "Pick out" and "Unstack," outperforming GR00T by a wide margin.
- XArm + XHand (Dexterous Hand): Since no dexterous hand data was used during pre-training, this setup tested true embodiment transfer. villa-X outperformed specialized baselines like GR-1 and GR00T, hitting an 84% success rate on seen pick-and-place tasks and 68% on unseen variations.
Limitations and Future Work
While the latent expert is highly effective at future planning, the authors note that its full generative potential is not yet fully exploited. Currently, it outputs a single deterministic plan path.
For future work, they propose integrating a VLM-based critic module. This would allow the latent expert to sample multiple diverse trajectory plans, while the critic rejects any plans that fail to align with the high-level language instructions before they are sent to the robot's physical controllers.
Summary updated