TL;DR
LaST1.0 is a unified World-Action Model that shifts the paradigm from predicting full future video frames to "control-centric" latent prediction. By focusing on compact, geometry-aware future targets from wrist-camera views and employing a novel 4D spatiotemporal attention mechanism, it achieves state-of-the-art performance and robust generalization across complex robotic manipulation tasks.
What Problem Does This Paper Address?
Traditional Vision-Language-Action (VLA) models often struggle with fine-grained manipulation because they lack an internal "understanding" of physical dynamics. To solve this, researchers developed World-Action Models (WAMs) that predict future video frames as an auxiliary task.
However, current WAMs are often "observation-complete," meaning they try to reconstruct every pixel of a future scene. This wastes massive model capacity on irrelevant details like background textures or lighting changes that don't affect the robot's next move. Furthermore, standard vision encoders (like ViTs) used in these models often prioritize high-level semantics over the tiny, localized geometric cues essential for precise grasping and assembly.
Methods
LaST1.0 introduces a "control-centric" architecture designed to focus only on what matters for the task:
- Geometry-Aware Latent Targets: Instead of predicting pixels, the model predicts compact latent representations derived from future wrist-camera views. It uses a frozen teacher model (VGGT-Ω) to generate these targets, which capture the evolving geometry of robot-object interactions.
- Encoder-Free Visual Interface: To avoid losing fine-grained details, LaST1.0 bypasses heavy pre-trained vision encoders. Instead, it uses a lightweight patch embedder to project raw image patches directly into the Transformer backbone.
- LaST-Attention & 4D-RoPE: The authors introduce a structured attention mechanism that uses 4D Rotary Positional Embeddings (RoPE). This allows the model to reason across four distinct dimensions: token sequence, image height, image width, and relative time.
- Task-Routed Mixture-of-Experts (MoE): The backbone features specialized experts for context understanding, latent dynamics modeling, and action generation, allowing the model to scale its capacity without increasing the computational cost per token during inference.
Datasets
The model was pre-trained on a massive corpus of ~3.9K hours of robotic data (255k trajectories) from sources like AGIBot and GenRobot. Evaluation was conducted on:
- Simulation: RLBench (10 tasks), LIBERO (40 tasks), and LIBERO-Plus (for out-of-distribution generalization).
- Real-World: Five multi-stage tasks (e.g., "Assemble Truck Toy," "Parcel-Box Packing") using different robot embodiments, including parallel grippers and dexterous hands.
Key Insights and Ideas
- Wrist-World Focus: The paper demonstrates that supervising the model with future wrist views is more effective than using external views. This "centers" the model's predictive capacity on the immediate interaction region where control happens.
- Geometry over Appearance: Ablations show that predicting geometry-aware latents leads to much higher success rates (92.0%) compared to predicting appearance-oriented VAE latents (87.4%).
- Training-Only Regularization: The latent prediction branch is only used during training. At inference time, it can be omitted, allowing for efficient closed-loop control without the overhead of predicting the future.
- The Power of Raw Patches: By letting the backbone see raw patches directly (encoder-free), the model maintains a stronger focus on manipulated objects, as seen in the attention map visualizations.
Key Results
LaST1.0 sets a new bar for performance across several benchmarks:
- RLBench: Achieved a 92.0% average success rate, outperforming the previous state-of-the-art (LaST0) by over 10 percentage points.
- LIBERO: Reached a leading 99.0% success rate.
- Generalization: On the challenging LIBERO-Plus benchmark, it showed superior robustness to sensor noise and background changes, significantly outperforming baselines that rely on standard video generation.
- Efficiency: Despite its complexity, the model maintains a fast inference time of 54.7ms on a single RTX 4090, making it highly practical for real-time robotics.
Limitations and Future Work
While highly effective, LaST1.0 relies on a frozen geometric teacher (VGGT-Ω) to provide supervision. If the teacher has blind spots in certain environments, the student model might inherit them. Additionally, while the model generalizes well to visual shifts, extreme changes in object geometry or physics (e.g., deformable objects) remain a frontier for future research. The authors suggest that integrating more diverse modalities, such as tactile feedback, could further enhance the model's "world-sensing" capabilities.
Summary updated