FlowBench: Counterfactual Imagination Training for Spatial Intelligence

Yifan Liu, Baihan Zhang, Khai Loong Aw, Klemen Kotar, Daniel L. K. YaminsarXiv 2026 · arXiv

TL;DR

FlowBench is a massive new resource containing 428,000 examples designed to train and evaluate Counterfactual Spatial Imagination (CSI)—the ability of AI to predict how a scene's spatial layout changes after a hypothetical object motion. The research reveals a significant "imagination gap" in current frontier models but demonstrates that large-scale training in simulation can lead to powerful "sim-to-real" transfer for robotic spatial reasoning.

What Problem Does This Paper Address?

While modern Vision-Language Models (VLMs) excel at describing what they see, they struggle to anticipate what would happen if the world changed. Humans do this effortlessly: we can mentally "see" if moving a couch will block a doorway before we ever touch it. This ability is critical for robotic planning and navigation.

The authors identify a missing capability they call Counterfactual Spatial Imagination (CSI). Existing benchmarks mostly test static perception or camera movement in static scenes. FlowBench addresses the gap where a model must infer the spatial consequences of object motions (e.g., "If I move this box 1 meter left, will it hide the cup?") without actually seeing the resulting image.

The concept of Counterfactual Spatial Imagination (CSI) and the "imagination gap" observed in current models.

Methods

The researchers formalized CSI as a visual question-answering task: given an initial image ($I_0$), a hypothetical motion ($t_m$), and a spatial question ($t_q$), the model must predict the answer ($t_a$) for the resulting (unseen) scene.

To generate data at scale, they developed a fully automatic pipeline:

  1. Positioning: Automatically finding informative camera views in simulated environments.
  2. Planning: Designing "answer-changing" motions (e.g., moving an object specifically to cause an occlusion) to ensure the dataset isn't trivial.
  3. Querying & Annotating: Using simulator metadata (AI2-THOR) to get ground-truth answers, optical flow, and depth maps. They also used VLMs to rephrase questions for linguistic variety and generate reasoning traces.

For real-world data, they mined existing robot manipulation videos, treating the observed robot actions as "hypothetical" motions and using the final video frames as the ground-truth outcomes.

Datasets

The authors introduced FlowBench, which consists of two primary components:

  • FlowBench-Sim: 420,000 training questions across 10,000 procedurally generated indoor houses. It covers eight categories including Occlusion, Feasibility, and Proximity.
  • FlowBench-Real: 8,575 training and 1,083 test questions derived from five real-world robot datasets (e.g., Bridge V2, DROID). It focuses on tabletop manipulation and includes a "Depth Order" category.
FlowBench covers diverse simulated home scenes and real-world tabletop robot manipulation tasks.

Key Insights and Ideas

  • The Imagination Gap: Even frontier models like GPT-6 Astra and Claude-Fable-5.1 exhibit an "imagination gap"—they perform significantly worse when they have to imagine a motion's outcome compared to when they are shown the resulting image. For Qwen3-VL-8B, this gap was a staggering 13 percentage points.
  • Sim-to-Real Power: One of the most surprising findings was that fine-tuning a model only on simulated data (FlowBench-Sim) led to massive gains on real-world images (+16.9 points for Qwen3-VL-8B). This suggests that spatial reasoning principles learned in "clean" simulations generalize well to the messy real world.
  • Models vs. Humans: On real-world tabletop scenes, frontier models actually outperformed humans in metric judgments (like estimating centimeter-level distances). Humans, conversely, tended to answer as if nothing had moved, failing to update their mental map of the scene.
  • World Models are Improving but Not Sufficient: Training world models on FlowBench improved their ability to predict optical flow. However, using these models to "imagine" a frame before answering a question (the "Imagine, then Answer" approach) actually hurt performance because current generative models lose the fine details (like small markers) that the questions depend on.

Key Results

  • VLM Improvement: Fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raised its accuracy by 29.5 percentage points in simulation and 16.9 points on real images.
  • World Model Gains: Training on FlowBench improved text-to-flow prediction for all tested model families. The unified world model $\Psi$-0.6 saw its accuracy rise from 52.1% to 61.7%, surpassing modular perception pipelines.
  • Frontier Performance: GPT-6 Astra reached 89.9% accuracy on FlowBench-Real-Test, but still trailed its own "observed" performance (93.0%), highlighting that imagination remains a bottleneck even for the strongest models.

Limitations and Future Work

The authors acknowledge several constraints:

  • Motion Complexity: Currently, motions are limited to single rigid translations. Future versions should include rotations, deformations (like folding a cloth), and multi-step interactions.
  • Metric Perception in Sim: Models (and humans) found the simulated scenes harder than real ones, likely due to the difficulty of judging distances in 3D renderings without familiar scale cues.
  • Faithful Generation: For world-model-assisted reasoning to work, generative models need to become much more "faithful," preserving the identity and precise location of all objects in the scene.

FlowBench provides a foundational resource for moving AI from static observers to agents that can "think ahead" in 3D space.

Summary updated

Abstract

Humans imagine scene changes under hypothetical motions, an ability central to spatial reasoning and robotic planning. However, this capability remains underexplored in vision-language models and world models. We formalize this capability as Counterfactual Spatial Imagination (CSI) and introduce FlowBench, a resource for training and evaluating both, with 428K training examples spanning simulated and real-world scenes, household environments and tabletop manipulation, and diverse question types. A scalable, fully automatic pipeline pairs each hypothetical motion with spatial questions and, in simulation, exact outcome frames and optical flow. For VLMs, proprietary models exhibit an imagination gap, performing worse when motion outcomes must be inferred than when observed, and open-source models perform near chance; fine-tuning Qwen3-VL-8B on FlowBench-Sim-Train raises its accuracy by 29.5 percentage points in simulation and 16.9 on real images, a strong sim-to-real transfer. For world models, training on FlowBench-Sim improves text-to-flow prediction for all four model families we test, raising the accuracy of Psi-0.6 from 52.1% to 61.7%. FlowBench thus serves as a training and evaluation resource for both. Code and datasets are released at https://github.com/liuyifan22/FlowBench.
BibTeX
@misc{liu2026flowbench,
  author = {Yifan Liu and Baihan Zhang and Khai Loong Aw and Klemen Kotar and Daniel L. K. Yamins},
  title = {{FlowBench: Counterfactual Imagination Training for Spatial Intelligence}},
  year = {2026},
  eprint = {2611.03937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03937},
}
Figure