Thinking in Streams: Proactive Spatial Reasoning with Vision-Language Models

Junyoung Hong, Minsu Kim, Jaehyun Kim, Jeonghwan Cho, Jiwoo Lee, Seon Joo KimarXiv 2026 · arXiv

TL;DR

This paper introduces ProSpace, a framework for "proactive spatial reasoning" where vision-language models (VLMs) must monitor a video stream and autonomously decide when a specific spatial condition is met. The authors contribute a new 9-task benchmark (ProSpace-Bench) and ProSpace-VLM, a geometry-aware model that uses its internal KV cache as a persistent spatial memory by injecting explicit 3D geometric encodings.

What Problem Does This Paper Address?

Most current evaluations of vision-language models are "reactive": a model is given a video clip and asked a question about it at a fixed point in time. However, real-world embodied agents (like robots or smart assistants) need to be proactive. They receive a request in advance—e.g., "Tell me when you enter the laundry room"—and must continuously process a stream of visual data to determine the exact moment to respond, or remain silent if the condition is never met.

The challenge lies in maintaining an updated "mental model" of space across changing viewpoints. Current VLMs often struggle with this because they lack a mechanism to explicitly represent and store 3D spatial relationships over long sequences, leading to premature or missed responses.

Figure 1. ProSpace brings spatial reasoning to streaming video in embodied environments, where models think in streams to decide when to respond to spatial requests. This requires maintaining and updating spatial knowledge across changing viewpoints.

Methods

The authors propose ProSpace-VLM, a streaming architecture designed for efficient spatial reasoning:

  1. Causal Geometry Encoding: For every incoming frame, a geometry encoder (CUT3R) estimates a 3D point map and camera pose. These are converted into Fourier-encoded geometric features.
  2. KV Cache as Spatial Memory: Instead of adding a separate memory module, the model injects these 3D encodings directly into the visual tokens at early-to-middle layers of the VLM decoder. This transforms the standard Key-Value (KV) cache into a spatially grounded memory that accumulates history as the camera moves.
  3. Voxel-Based Token Selection: To keep the KV cache manageable during long streams, the model uses a voxel-based selection strategy. It keeps the most recent frames in full but thins out older frames by retaining only one token per 3D voxel, ensuring the memory remains anchored in a consistent coordinate frame.
  4. Streaming Decision Head: The model is trained to predict two special tokens: wait and respond. It only triggers a response when the probability of the respond token crosses a certain threshold.
The ProSpace-VLM architecture injects explicit 3D geometry into the KV cache to enable persistent spatial memory.

Datasets

The paper introduces two major resources:

  • ProSpace-Bench: A benchmark consisting of 3,636 episodes across 242 real-world indoor scenes. It features nine tasks categorized into three levels:
  • Object-Level: Finding, re-finding, and determining the direction of objects.
  • Camera-Level: Detecting ego-rotation, origin return, and visual homing.
  • Scene-Level: Identifying room transitions, entries, and revisits.
  • ProSpace-Train: A large-scale training set of 100,000 streams with supervision for response timing and abstention (knowing when not to answer).

The data is sourced from established 3D datasets like ScanNet, ScanNet++, CA-1M, and Matterport3D.

Overview of the nine tasks in ProSpace-Bench across object, camera, and scene levels.

Key Insights and Ideas

  • The Human-AI Gap: There is a massive performance gap between humans and current VLMs. While humans achieve ~83% timely response rates on the benchmark, even the best zero-shot proprietary models like GPT-4o hover around 24.5%.
  • Explicit Geometry Matters: Simply fine-tuning a VLM on streaming data isn't enough. Injecting explicit 3D coordinates and camera poses into the internal layers significantly improves the model's ability to "remember" where objects are once they leave the field of view.
  • KV Cache is Sufficient Memory: The paper demonstrates that we don't necessarily need complex external memory architectures; the existing KV cache in Transformers can serve as an effective spatial map if the tokens are properly grounded in 3D space.

Key Results

  • Performance Boost: ProSpace-VLM achieved a 71.8% timely response rate and 87.3% correct abstention, significantly outperforming fine-tuned general-purpose baselines (like Qwen2.5-VL) which scored 61.7% and 78.7% respectively.
  • Object Permanence: Probing the model's internal states showed that ProSpace-VLM maintains a much more accurate "belief" about the direction of an object after it is no longer visible compared to standard VLMs.
  • Efficiency: The voxel-based token selection allowed the model to maintain real-time inference speeds (6.7 FPS) while nearly halving peak memory usage on long video streams (1,024 frames).
Breakdown of response timing, showing that ProSpace-VLM significantly reduces premature and missed responses compared to baselines.

Limitations and Future Work

  • Rule-Based Construction: The benchmark episodes are constructed using automated rules (e.g., based on camera pose and object visibility), which might not capture the full nuance of natural language spatial requests.
  • Limited Scope of Action: The current model only decides when to speak. It does not yet integrate with high-level reasoning or physical robot control (e.g., "Go find the bag").
  • Future Directions: The authors suggest exploring how these proactive models can be integrated into full embodied agents that can actively move to verify spatial conditions.

Summary updated

Abstract

Recognizing when a requested spatial condition holds in streaming video can require maintaining and updating spatial knowledge across changing viewpoints. We study this capability through proactive spatial reasoning, where a model receives a query in advance and determines whether and when to respond as the stream unfolds. We introduce ProSpace-Bench, comprising nine tasks across object, camera, and scene reasoning, and ProSpace-Train for supervised training. To support spatial decisions, we propose ProSpace-VLM, which injects explicit geometric encodings estimated causally from incoming frames into intermediate VLM layers, using the existing KV cache as spatial memory and spatial token selection for efficient streaming. Experiments reveal a substantial gap between VLMs and humans, while ProSpace-VLM improves timely responses and correct abstention over fine-tuned baselines. Analyses examine how ProSpace-VLM represents previously observed spatial information across changing viewpoints.
BibTeX
@misc{hong2026thinking,
  author = {Junyoung Hong and Minsu Kim and Jaehyun Kim and Jeonghwan Cho and Jiwoo Lee and Seon Joo Kim},
  title = {{Thinking in Streams: Proactive Spatial Reasoning with Vision-Language Models}},
  year = {2026},
  eprint = {2611.25865},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25865},
}
Figure