Pros4D: Modeling Evolving 4D Worlds in Latent Space for Dynamic Spatial Reasoning

Rongxin Gao, Dongxuan Liu, Chu Li, Chengyang Ji, Jialing Huang, Runyu Chen, Kairun Wen, Chenxin Li, Jingrui Ye, Haosong Zhen, Xinghao Ding, Yue Huang, Yuzhi Huang, Xiaodong Tu, Jingyan Jiang

TL;DR

Pros4D is a framework that enables Vision-Language Models (VLMs) to perform dynamic spatial reasoning by modeling the world in a shared 4D latent space. By grounding internal "latent states" with both visual and geometric supervision, the model can retrospectively understand observed video and prospectively predict future object trajectories and spatial relationships without needing external geometric tools at inference.

What Problem Does This Paper Address?

While current Vision-Language Models (VLMs) have made strides in understanding static scenes, they often struggle with dynamic spatial reasoning—understanding how 3D environments evolve over time.

Most existing methods focus on "retrospective" understanding (what just happened in the video). However, embodied agents like robots or self-driving cars need "prospective" reasoning: predicting what will happen next. Current approaches typically rely on either:

  1. External geometric tools: These add inference latency and complexity.
  2. Textual descriptions: Verbalizing continuous motion is "lossy" and fails to capture fine-grained 3D dynamics.

The authors of Pros4D argue that models need an internal "mental model"—a shared latent state that preserves observed dynamics and remains predictive beyond the input video.

Methods

The core of the paper is the Pros4D framework, which consists of three main pillars:

  1. 4D Latent State Reasoning (4D-LSR): Instead of relying purely on text, the model generates sequences of continuous hidden vectors (latent tokens) within its reasoning chain. It uses <latent_observed_state> tokens to compress past dynamics and <latent_future_state> tokens to autoregressively "roll forward" the scene's evolution into the future.
  2. 4D Latent Alignment (4DLA): This is the training strategy used to ground these latent tokens. It employs a dual-supervision approach:
  3. Visual-Semantic Alignment: Ensures latent tokens capture the appearance and identity of the queried entity.
  4. Geometric Alignment: Uses a frozen geometry encoder (Pi3X) to supervise intermediate layers of the VLM, forcing the latent states to preserve 3D structural information like distance and orientation.
  5. Pros4D-Engine: An automated data pipeline that generates training data from raw monocular video. It uses SAM3 for tracking, Depth Anything 3 for depth estimation, and a physics-guided reasoner to construct high-quality Q&A pairs grounded in 3D geometry.

Datasets

  • Pros4D-40K: A newly synthesized training dataset containing 40,000 examples with reasoning traces.
  • Pros4D-Bench: A new evaluation benchmark featuring 3,000 QA pairs across eight spatial categories (e.g., Camera-Object Distance, Inter-Object Distance, Scene Trajectory).
  • Generalization Benchmarks: The model was also tested on several external benchmarks including Dyn-Bench, VLM4D, OpenX-VQA (robotics manipulation), and STRIDE-QA-Bench (autonomous driving).

Key Insights and Ideas

  • The Power of the Latent "Think" Block: By interleaving latent tokens within a <think> tag, the model performs internal computation that isn't restricted by the vocabulary of a natural language. This allows it to "imagine" 3D motion more accurately than text alone.
  • Geometry as a Teacher, Not a Crutch: A major "aha" moment is that geometric supervision is only needed during training. Once the latent space is aligned with 3D structures, the model can reason about depth and motion at inference time using only raw video pixels.
  • Unified Retrospection and Prospection: The paper demonstrates that the same latent representation used to understand the past is highly effective for predicting the future. The "observed" states provide the momentum and context necessary for the "future" states to remain physically plausible.

Key Results

The results show that Pros4D significantly elevates the spatial intelligence of base VLMs:

  • Pros4D-8B (based on Qwen3-VL) achieved a 62.9% average accuracy on Pros4D-Bench, outperforming its backbone by 16.5 percentage points.
  • Future Prediction: On prospective tasks, the model saw an 18.8-point improvement over the baseline, proving the effectiveness of the autoregressive latent rollout.
  • Robotics & Driving: In the OpenX-VQA (robotics) benchmark, Pros4D-3B improved average accuracy by 20.9 points, particularly excelling at anticipating the "next action" in a manipulation task (+52.6 points).
  • Ablation Studies: Removing geometric alignment led to a significant drop in accuracy, confirming that "seeing" the 3D structure is vital for reasoning about motion.

Limitations and Future Work

  • Geometric Teacher Dependency: The model's spatial accuracy is currently capped by the quality of the automated 3D labels provided by the Pros4D-Engine (e.g., errors in monocular depth estimation).
  • Short Horizons: The current work focuses on relatively short-term future predictions (1-3 seconds). Extending this to longer horizons in complex, unpredictable environments remains a challenge.
  • Computational Overhead: While more efficient than external tool-calling, generating 16 latent tokens per state does increase the number of tokens processed during inference compared to a standard text-only VLM.

Summary updated

Abstract

Dynamic spatial reasoning from monocular video enables vision-language models (VLMs) to understand how 3D environments evolve over time, supporting physical world understanding and embodied intelligence. Existing efforts primarily address observed-dynamics understanding, whereas agents in dynamic environments also require future-dynamics prediction beyond the observation boundary. Moreover, external geometric processing and textual descriptions can increase inference complexity or lose information when representing continuous dynamics. These limitations motivate a shared internal state that preserves observed dynamics and remains predictive beyond the input video. Accordingly, we present Pros4D, a framework for modeling evolving 4D worlds in latent space that unifies retrospective understanding of observed-dynamics with prospective reasoning for future-dynamics prediction. Specifically, Pros4D-Engine derives geometry-grounded supervision and constructs paired examples for both tasks, while Pros4D-Bench evaluates them across eight spatial task categories spanning dynamic cameras, objects, and scenes. At the model level, Pros4D compresses observed dynamics into latent observed states and autoregressively rolls them forward into latent future states. 4D Latent Alignment (4DLA) grounds these states through complementary visual-semantic and geometric supervision, while joint language supervision connects them to task reasoning. Experiments across benchmarks show consistent improvements over strong baselines in both capabilities.
BibTeX
@misc{gao2026pros,
  author = {Rongxin Gao and Dongxuan Liu and Chu Li and Chengyang Ji and Jialing Huang and Runyu Chen and Kairun Wen and Chenxin Li and Jingrui Ye and Haosong Zhen and Xinghao Ding and Yue Huang and Yuzhi Huang and Xiaodong Tu and Jingyan Jiang},
  title = {{Pros4D: Modeling Evolving 4D Worlds in Latent Space for Dynamic Spatial Reasoning}},
  year = {2026},
  eprint = {2611.19803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19803},
}

Loading related content...

Figure