Flex-$π$: A Multi-Stream World-Action Model with Compute Flexibility

Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter FoxarXiv 2026 · arXiv

TL;DR

FLEX-$\pi$ is a 6B-parameter world-action model (WAM) that achieves exceptional demonstration efficiency and generalization by learning to jointly predict future RGB images, 3D geometry, and object semantics. By leveraging a "free lunch"—the fact that frozen video-generation VAEs can encode 3D pointmaps almost losslessly—FLEX-$\pi$ introduces zero sensor or pre-training overhead, and features a flexible architecture that lets users trade inference speed for performance at deployment time.

What Problem Does This Paper Address?

Generalist robot policies are shifting from Vision-Language-Action models (VLAs) toward World-Action Models (WAMs), which predict future observations alongside actions. However, most current WAMs only predict future RGB pixels. While RGB prediction inherits strong spatiotemporal priors from internet-scale video models, pixel reconstruction primarily captures surface appearance details rather than the explicit 3D geometry and object semantics crucial for precise physical manipulation.

Directly supervising geometry and semantics usually comes at a steep price: requiring extra physical sensors, training new visual priors from scratch, or sacrificing inference speed. FLEX-$\pi$ addresses this challenge by incorporating 3D and semantic streams into a WAM without incurring any of these costs.

Methods

FLEX-$\pi$ utilizes a Mixture-of-Transformers (MoT) backbone consisting of a 5B-parameter shared visual trunk and a 1B-parameter action expert.

FLEX-$\pi$ multi-stream architecture overview

The core architecture relies on three key algorithmic components:

  1. The Shared Latent Space "Free Lunch": The authors discovered that a frozen VAE from a pre-trained video generation model (Wan-2.2), trained purely on RGB data, can encode and reconstruct 3D pointmaps almost losslessly. FLEX-$\pi$ exploits this by projecting both RGB images and 3D pointmaps (generated off-the-shelf via Depth Anything 3) into the exact same latent space. Object semantics are supplied via a frozen DINOv3 encoder.
  2. Visual Stream Dropout & Cross-Modality Forcing: During training, input modalities are randomly dropped out with a 50% probability. Crucially, the model is still forced to denoise and predict all future streams (RGB, 3D, and DINO) regardless of what was provided as input. This forces the internal representations of appearance, geometry, and semantics to become mutually predictive.
  3. Compute Flexibility: Because the action tokens use one-way attention and do not dictate the visual generation path, any visual stream can be dropped at inference time. This allows a single trained checkpoint to run in a fast, "action-only" mode or a high-accuracy, "full joint generation" mode.
Detailed FLEX-$\pi$ architecture diagram

Datasets

  • Pre-training: ~500 hours of bimanual manipulation data from 100 tasks in the AGIBOT World-Beta dataset.
  • Simulation Evaluation: RoboTwin (50 tasks for data-scaling and Pareto analyses), LIBERO (40 tasks across 4 suites), and LIBERO-Plus (testing robustness against 7 categories of visual, spatial, and language perturbations).
  • Real-World Evaluation: Collected on a bimanual YAM robot across 5 dexterous tasks including Kitchen Organization, Self-Repair Gripper (requiring $\pm0.25$ mm screwdriver clearance), and Soft-Bag Zipping (manipulating deformable objects).

Key Insights and Ideas

  • The VAE Geometry Prior: A standard video VAE contains latent spatial structures robust enough to represent metric 3D pointmaps without any pointmap-specific fine-tuning.
  • Imagination as a Regularizer: Cross-modality forcing acts like a masked autoencoder for physics. By forcing the model to "imagine" 3D geometry from RGB inputs (and vice versa) during training, the policy develops a deeply grounded internal representation that drastically improves action execution—even if you turn off the visual imagination streams during deployment to save compute.

Key Results

  • Real-World Performance: FLEX-$\pi$ outperformed strong baselines ($\pi_{0.5}$, ManiFlow, Fast-WAM) by 2-6$\times$ in success rates on complex bimanual tasks. It achieved a 76% success rate on the sub-millimeter precision Self-Repair Gripper task, where $\pi_{0.5}$ failed completely.
Real-world task completion results across five bimanual tasks
  • Inference Speed: In action-only mode, FLEX-$\pi$ runs at 60ms per call on a single RTX 5090—faster than $\pi_{0.5}$—while maintaining superior accuracy due to its multi-stream training. Enabling full joint generation increases latency to ~193ms but yields an additional 13% boost in success rates.
  • Data Efficiency: On the RoboTwin benchmark with a limited budget of 50 demonstrations per task, FLEX-$\pi$ outperformed the strongest WAM baseline by 1.9$\times$.
  • Simulation Benchmarks: Achieved a 99.2% overall success rate on LIBERO, matching or beating state-of-the-art models pre-trained on up to 76$\times$ more data.

Limitations and Future Work

  • Convergence Time: Due to the complexity of optimizing multiple streams and cross-modality forcing, FLEX-$\pi$ requires more training epochs to converge compared to single-stream models (at least 10 epochs of real-world fine-tuning).
  • Inference Latency in Full Mode: Generating all visual futures jointly (~193ms) remains slower than pure VLA baselines, making it less ideal for highly reactive, high-frequency control loops unless run in action-only mode.
  • Semantic Reasoning: While geometrically grounded, the model still lags slightly behind massive VLAs on pure semantic generalization axes (like complex language prompt variations in LIBERO-Plus), suggesting that pairing FLEX-$\pi$ with a larger vision-language backbone is a promising future direction.

Summary updated

Abstract

World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-$π$, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7$\times$ on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than $π_{0.5}$. Our project website: https://flex-pi.github.io/
BibTeX
@misc{yan2026flex,
  author = {Ge Yan and Jinghao Liu and Yuzhi Fan and Lei Cai and Minwen Liao and Jesse Zhang and Dieter Fox},
  title = {{Flex-$π$: A Multi-Stream World-Action Model with Compute Flexibility}},
  year = {2026},
  eprint = {2608.10860},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.10860},
}
Figure