ForeAct: Geometry-Guided Action Learning from World Representations

Wei Lu, Jiaxi Zhang, Yunheng Wang, Taowen Wang, Yuetong Fang, Zhenhui Zhang, Shuning Zhang, Zizhao Yuan, Yixiao Feng, Zecui Zeng, Renjing XuarXiv 2026 · arXiv

TL;DR

ForeAct repurposes pretrained video world models as efficient, single-pass backbones for robotic manipulation, eliminating the need for slow, iterative video generation. By integrating learnable "geometry tokens" supervised by next-frame depth during training, it creates a future-aware, spatially grounded representation that achieves state-of-the-art performance with low inference latency.

What Problem Does This Paper Address?

Effective robotic manipulation requires a model to understand two things simultaneously: the 3D structure of the scene (geometry) and how that scene changes over time (dynamics).

Current approaches usually pick one at the expense of the other. World Action Models (WAMs) are great at dynamics because they are trained to predict future video frames, but they are computationally expensive. They typically rely on iterative diffusion processes—essentially "denoising" an image over many steps—which is too slow for real-time, closed-loop control. Furthermore, their internal representations often lack explicit spatial structure. On the other hand, geometry-centric policies use specialized 3D backbones that understand space well but often struggle to anticipate how that space will evolve during a complex interaction.

ForeAct aims to bridge this gap by creating a unified representation that is both geometry-aware and fast enough for real-world deployment.

Methods

The core innovation of ForeAct is how it "repurposes" a pretrained video world model (specifically the Cosmos-Predict2-2B model). Instead of using the model to generate future frames through a slow diffusion process, ForeAct treats it as a feature extractor in a single forward pass.

Key components include:

  • Single-Pass World Representation: The model processes the "clean" latent of the current observation just once. This extracts the rich spatiotemporal priors already present in the pretrained world model without the overhead of iterative denoising.
  • Learnable Geometry Tokens: The authors insert a small set (typically 8) of learnable tokens into the model's input stream alongside visual and text tokens.
  • Bidirectional Self-Attention: These geometry tokens interact with visual tokens through the model's attention layers. This allows the geometric information to "shape" the visual features used for action prediction.
  • Predictive Geometry Supervision: During training, these tokens are decoded into the relative depth map of the next frame (see Figure 3). This auxiliary task forces the model to learn a representation that anticipates future spatial configurations. Crucially, this geometry head is discarded at inference time, leaving a streamlined model that only requires RGB input.
The ForeAct architecture showing the single-pass forward flow and the auxiliary geometry branch used during training.

Datasets

The researchers evaluated ForeAct across three major domains:

  1. LIBERO: A standard simulation benchmark for lifelong robotic learning, featuring four suites (Spatial, Object, Goal, and Long) that test generalization across different task structures.
  2. SimplerEnv-Bridge: A simulation environment focused on visual-matching manipulation using a WidowX robot, testing robustness to changes in appearance and object placement.
  3. Real-World Manipulation: The authors conducted 90 rollouts on a physical Franka Emika Panda robot across six tabletop tasks, including precise alignment (Stack Bowls) and forceful interaction (Sandbag-to-Scale).

Key Insights and Ideas

  • Generative Priors are Sufficient: You don't actually need to generate a future video to benefit from a world model's knowledge. The internal features of a single forward pass are stable and rich enough for high-quality control.
  • Geometry as a Regularizer: Supervising the model with next-frame depth acts as a powerful regularizer. It grounds the abstract features of the video model in physical reality, making the resulting actions more precise.
  • The "Goldilocks" Token Count: Interestingly, increasing the number of geometry tokens beyond a certain point (8 tokens) actually decreased performance (see Figure 7). A compact set of tokens provides a sufficient bottleneck for effective geometric supervision without over-parameterizing the interface.
  • Bidirectional Interaction is Key: The model performs significantly better when geometry tokens can influence visual features (and vice versa) compared to a one-way information flow. This suggests that "spatial reasoning" needs to be deeply integrated into the visual backbone rather than just added as a sidecar.

Key Results

ForeAct demonstrates a superior balance of accuracy and speed compared to existing Vision-Language-Action (VLA) models and WAMs:

  • Simulation Success: It achieved a 96.6% average success rate on LIBERO, outperforming larger models like Cosmos Policy (98.5% but much slower) and significantly beating standard VLAs like OpenVLA (76.5%).
  • Real-World Performance: In 90 real-world trials, ForeAct reached a 74.4% success rate, notably outperforming π0 (57.8%) and Fast-WAM (64.4%).
  • Efficiency: ForeAct operates with an inference latency of 168ms on an NVIDIA H20 GPU. As shown in Figure 6, it sits in a "faster and better" regime compared to most contemporary world action models.
Success rate vs. inference latency on LIBERO. ForeAct (purple circle) shows a high success rate with significantly lower latency than other World Action Models.

Limitations and Future Work

  • Long-Horizon Reasoning: While ForeAct performs well on the LIBERO-Long suite (91.8%), this is still its lowest-scoring simulation category, suggesting that multi-stage task planning remains a challenge.
  • Single-Frame Input: The current model relies on a single current observation. The authors suggest that incorporating historical frames could further improve temporal consistency.
  • Dependence on Depth Estimators for Training: The training process requires high-quality relative depth targets, currently provided by an offline teacher model (MoGe-3). The quality of ForeAct is therefore partially bounded by the quality of the depth supervisor used during training.

Summary updated

Abstract

Robotic manipulation requires representations that capture both scene geometry and its evolution during interaction. Existing approaches have largely treated these two aspects separately. World action models capture such dynamics by coupling action prediction with future video generation, but their reliance on iterative diffusion or rectified-flow inference leads to high computational cost, while their representations lack explicit spatial structure. Explicitly modeling scene geometry typically requires auxiliary modules that increase both parameter count and computational cost. Conversely, geometry-centric policies primarily recover observed spatial structure and often lack explicit supervision for anticipating future geometric changes during interaction. To address these limitations, we introduce ForeAct, which repurposes a pretrained video world model as a single-pass representation backbone for low-latency control. ForeAct directly processes the clean latent of the current observation in one forward pass, eliminating future-frame synthesis and iterative denoising while retaining the backbone's pretrained spatiotemporal priors. Rather than introducing a separate geometry backbone, ForeAct uses learnable tokens supervised by next-frame relative depth to shape the shared video representation through bidirectional self-attention. This compact interface turns the spatial knowledge already encoded in the backbone into a future-aware representation for action generation, with little parameter overhead. The lightweight geometry readout is used only during training and removed at deployment, enabling low-latency, geometry-aware control without depth input or an external geometry backbone. Without additional large-scale pretraining, ForeAct achieves 96.6% average success on LIBERO, 63.5% on SimplerEnv-Bridge, and a 74.4% success rate across 90 real-world rollouts.
BibTeX
@misc{lu2026foreact,
  author = {Wei Lu and Jiaxi Zhang and Yunheng Wang and Taowen Wang and Yuetong Fang and Zhenhui Zhang and Shuning Zhang and Zizhao Yuan and Yixiao Feng and Zecui Zeng and Renjing Xu},
  title = {{ForeAct: Geometry-Guided Action Learning from World Representations}},
  year = {2026},
  eprint = {2611.11500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11500},
}
Figure