LaST1.0: A Unified Latent World-Action Model with Control-Centric SpatioTemporal Reasoning

Hao Chen, Xuheng Zhang, Zhuangzhe Wu, Jiaming Liu, Shaobo Zhang, Jialin Gao, Zhonghao Yan, Di Wu, Sixiang Chen, Peng Jia, Chi-Wing Fu, Shanghang Zhang, Pheng-Ann HengarXiv 2026 · arXiv

TL;DR

LaST1.0 is a unified World-Action Model that shifts the paradigm from predicting full future video frames to "control-centric" latent prediction. By focusing on compact, geometry-aware future targets from wrist-camera views and employing a novel 4D spatiotemporal attention mechanism, it achieves state-of-the-art performance and robust generalization across complex robotic manipulation tasks.

What Problem Does This Paper Address?

Traditional Vision-Language-Action (VLA) models often struggle with fine-grained manipulation because they lack an internal "understanding" of physical dynamics. To solve this, researchers developed World-Action Models (WAMs) that predict future video frames as an auxiliary task.

However, current WAMs are often "observation-complete," meaning they try to reconstruct every pixel of a future scene. This wastes massive model capacity on irrelevant details like background textures or lighting changes that don't affect the robot's next move. Furthermore, standard vision encoders (like ViTs) used in these models often prioritize high-level semantics over the tiny, localized geometric cues essential for precise grasping and assembly.

Methods

LaST1.0 introduces a "control-centric" architecture designed to focus only on what matters for the task:

  • Geometry-Aware Latent Targets: Instead of predicting pixels, the model predicts compact latent representations derived from future wrist-camera views. It uses a frozen teacher model (VGGT-Ω) to generate these targets, which capture the evolving geometry of robot-object interactions.
  • Encoder-Free Visual Interface: To avoid losing fine-grained details, LaST1.0 bypasses heavy pre-trained vision encoders. Instead, it uses a lightweight patch embedder to project raw image patches directly into the Transformer backbone.
  • LaST-Attention & 4D-RoPE: The authors introduce a structured attention mechanism that uses 4D Rotary Positional Embeddings (RoPE). This allows the model to reason across four distinct dimensions: token sequence, image height, image width, and relative time.
  • Task-Routed Mixture-of-Experts (MoE): The backbone features specialized experts for context understanding, latent dynamics modeling, and action generation, allowing the model to scale its capacity without increasing the computational cost per token during inference.
Figure 2. LaST1.0 Framework. (a) LaST1.0 employs an encoder-free unified backbone that takes projected multimodal context to jointly predict flow velocities for latent and action tokens. The latent targets are extracted via VGGT-Ω from future wrist views to enable interaction-centric geometry-aware reasoning. (b) Each decoder block in LaST1.0 features LaST-Attention, which independently rotates queries and keys across token sequence, image height, image width, and time axes to enhance spatiotemporal fusion. A subsequent Task-Routed MoE effectively expands model capacity. (c) Multimodal Tokens are represented in a 4D space with separate sequence, spatial and temporal coordinates, preserving their inherent structure.

Datasets

The model was pre-trained on a massive corpus of ~3.9K hours of robotic data (255k trajectories) from sources like AGIBot and GenRobot. Evaluation was conducted on:

  • Simulation: RLBench (10 tasks), LIBERO (40 tasks), and LIBERO-Plus (for out-of-distribution generalization).
  • Real-World: Five multi-stage tasks (e.g., "Assemble Truck Toy," "Parcel-Box Packing") using different robot embodiments, including parallel grippers and dexterous hands.

Key Insights and Ideas

  • Wrist-World Focus: The paper demonstrates that supervising the model with future wrist views is more effective than using external views. This "centers" the model's predictive capacity on the immediate interaction region where control happens.
  • Geometry over Appearance: Ablations show that predicting geometry-aware latents leads to much higher success rates (92.0%) compared to predicting appearance-oriented VAE latents (87.4%).
  • Training-Only Regularization: The latent prediction branch is only used during training. At inference time, it can be omitted, allowing for efficient closed-loop control without the overhead of predicting the future.
  • The Power of Raw Patches: By letting the backbone see raw patches directly (encoder-free), the model maintains a stronger focus on manipulated objects, as seen in the attention map visualizations.
Action-to-Vision attention maps showing how the encoder-free design focuses more precisely on task-relevant objects compared to ViT or VAE variants.

Key Results

LaST1.0 sets a new bar for performance across several benchmarks:

  • RLBench: Achieved a 92.0% average success rate, outperforming the previous state-of-the-art (LaST0) by over 10 percentage points.
  • LIBERO: Reached a leading 99.0% success rate.
  • Generalization: On the challenging LIBERO-Plus benchmark, it showed superior robustness to sensor noise and background changes, significantly outperforming baselines that rely on standard video generation.
  • Efficiency: Despite its complexity, the model maintains a fast inference time of 54.7ms on a single RTX 4090, making it highly practical for real-time robotics.
Table 2. Results on RLBench. All methods are trained in the multi-task setting, and we report average success rates (%).

Limitations and Future Work

While highly effective, LaST1.0 relies on a frozen geometric teacher (VGGT-Ω) to provide supervision. If the teacher has blind spots in certain environments, the student model might inherit them. Additionally, while the model generalizes well to visual shifts, extreme changes in object geometry or physics (e.g., deformable objects) remain a frontier for future research. The authors suggest that integrating more diverse modalities, such as tactile feedback, could further enhance the model's "world-sensing" capabilities.

Summary updated

Abstract

World-Action Models couple action learning with future prediction, enabling robotic policies to capture environmental dynamics for robust control. However, prevailing methods take a reconstruction-oriented approach to model future videos, allocating substantial capacity to appearance details that are incidental to control. In this work, we reformulate world modeling for manipulation as control-centric future prediction in latent space, and present LaST1.0, a unified latent World-Action Model. Rather than modeling an observation-complete video future, LaST1.0 predicts compact, geometry-aware latent targets derived from future wrist views, focusing the supervision on the evolving robot-object interactions required for fine-grained manipulation. To better support this control-centric paradigm, LaST1.0 revisits the visual input interface and attention mechanism of its unified architecture. It adopts a vision-encoder-free design with a lightweight patch embedder, exposing fine-grained visual cues directly to the backbone, and introduces LaST-Attention for structured spatiotemporal interactions across modalities. Evaluations across two robot embodiments, five challenging manipulation tasks, and three simulation benchmarks---RLBench, LIBERO, and LIBERO-Plus---demonstrate that LaST1.0 achieves state-of-the-art success rates and generalizes to novel scenarios. These results demonstrate the effectiveness of control-centric latent world modeling for robust manipulation.
BibTeX
@misc{chen2026last,
  author = {Hao Chen and Xuheng Zhang and Zhuangzhe Wu and Jiaming Liu and Shaobo Zhang and Jialin Gao and Zhonghao Yan and Di Wu and Sixiang Chen and Peng Jia and Chi-Wing Fu and Shanghang Zhang and Pheng-Ann Heng},
  title = {{LaST1.0: A Unified Latent World-Action Model with Control-Centric SpatioTemporal Reasoning}},
  year = {2026},
  eprint = {2611.23974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23974},
}
Figure