TL;DR
LaST0.5 is a generalist World Action Model that bridges the gap between global scene prediction and precise robotic control through "Interaction-Centric Latent Reasoning." By distilling global world representations into compact, interaction-focused abstractions and using an adaptive execution mechanism, it achieves state-of-the-art performance on major benchmarks and robust real-world bimanual manipulation.
What Problem Does This Paper Address?
World Action Models (WAMs) are the current "hot" paradigm in robotics. They work by predicting how a scene will evolve (world modeling) and then using that "imagined" future to decide what the robot should do. However, there is a fundamental mismatch: world models are optimized to predict global scene changes (like a ball rolling across a table), while manipulation requires a laser focus on fine-grained, local interactions (like the exact moment a gripper makes contact with a handle).
Directly transferring global representations to action prediction often results in "blurry" focus, where the model misses the critical interaction bottlenecks. Furthermore, world models can "hallucinate" physically impossible futures, leading to compounding errors when a robot blindly follows an imagined trajectory that has gone off the rails.
Methods
The authors propose LaST0.5, which introduces a hierarchical reasoning flow: Video → Interaction → Action. It uses a Mixture-of-Transformers (MoT) architecture comprising three specialized "experts":
- World Expert: A pretrained diffusion transformer (Cosmos) that forecasts future scene evolution.
- Latent Expert: The core innovation. It distills the World Expert's global predictions into a compact Interaction-Centric Latent Reasoning (ICLR) space. During training, this space is supervised by visual cues—2D curves for end-effector motion and bounding boxes for object affordances.
- Action Expert: Generates the final robot actions conditioned on both the global world state and the distilled interaction latents.
To handle hallucinations, they introduce Adaptive World-to-Action Transfer (AWAT). A value head predicts "progress values" for the imagined future. If the predicted progress stalls or regresses (indicating a failed "imagination"), AWAT truncates the action execution and triggers a fresh resampling of the world model.
Datasets
LaST0.5 was pretrained on a massive, heterogeneous corpus of over 14,000 hours of robotic data, including:
- Genrobot (UMI): 9,000+ hours of handheld gripper data.
- AgiBot & RoboCOIN: Bimanual teleoperation data.
- DROID: Single-arm teleoperation.
- EgoDex: Human egocentric manipulation videos.
- InternData-A1: Simulated and real data.
Evaluation was conducted on the LIBERO and RLBench simulation benchmarks, as well as real-world bimanual tasks using a Tianji Marvin robot.
Key Insights and Ideas
- Interaction as a Bridge: The most critical finding is that global scene dynamics are not enough for precise control. The "Latent Expert" acts as a necessary translator, focusing the model's attention on the "manipulation bottleneck"—the specific region where the robot meets the object.
- Visual Prompts as Latent Supervision: Using simple 2D curves and bounding boxes only during training to guide the latent space is incredibly effective. It forces the model to learn what "interaction" looks like without needing these prompts at inference time.
- Progress as a Sanity Check: AWAT turns a weakness of world models (hallucinations) into a trigger for closed-loop correction. By monitoring if an imagined task is actually "progressing," the system knows when to stop and look again.
Key Results
- SOTA Performance: LaST0.5 achieved a 99.5% average success rate on LIBERO and 88.0% on RLBench, significantly outperforming previous state-of-the-art models like $\pi_{0.5}$ and Cosmos Policy.
- Real-World Robustness: In bimanual tasks like "Parcel-Box Packing" and "Screw Fastening," it achieved a 63.8% average success rate, nearly double that of some baselines.
- Zero-Shot Generalization: The model showed strong adaptability to unseen objects, backgrounds, and spatial shifts, largely thanks to the rich physical priors learned during its massive pretraining.
Limitations and Future Work
While powerful, LaST0.5 still faces challenges in high-precision tasks. The authors noted failures in "Screw Fastening" due to millimeter-level misalignments that visual feedback alone couldn't resolve. Hardware limitations, such as insufficient grip force for heavy objects or mechanical resistance in zippers, also led to failures. Future work likely involves integrating tactile feedback or even more fine-grained spatial reasoning to handle these "tight-tolerance" scenarios.
Summary updated