ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence

Yihao Hu, Yitong Lin, Pan Wang, Zhihao Wen, Hang Wang, Yiwen Zhao, Deyuan Liu, Yutong Li, Mingda Wan, Yuchen Mao, Tao LinarXiv 2026 · arXiv

TL;DR

ChronoReplay is a novel post-training recipe that enables a single Vision-Language Model (VLM) to master diverse spatial tasks—categorical, metric, and temporal—simultaneously. By replaying heterogeneous spatial data in its original chronological order and balancing different supervision "families," it overcomes "interface lock-in" to achieve state-of-the-art results across multiple streaming spatial benchmarks.

What Problem Does This Paper Address?

Current spatial VLMs suffer from a phenomenon the authors call interface lock-in. When you train a model to answer categorical questions (e.g., "Is the chair on the left?"), it often loses the ability to perform metric estimation (e.g., "How many meters away is the chair?"), and vice versa. Improvements in one "interface" often come at the cost of degradation in others.

Furthermore, spatial intelligence in a streaming context (like a robot moving through a house) is inherently causal. An object's position relative to the observer depends entirely on the history of the observer's motion. If a model is trained on isolated snapshots or shuffled data, it loses the "temporal origin" that gives spatial coordinates their meaning.

Figure 1: Why reusable spatial state. Categorical preservation and trajectory supervision constrain complementary spatial distinctions. Causal Replay combines them in shared parameters. The illustrated readouts span categorical, metric, temporal, and reference-frame queries.

Methods

The authors propose ChronoReplay, which consists of two primary components:

  1. Causal Replay Construction: Instead of treating training data as independent question-answer pairs, ChronoReplay organizes data into chronologically ordered episodes. Each target answer is attached to its "causal prefix"—the exact sequence of observations leading up to that moment. This forces the model to learn how spatial state accumulates over time.
  2. Family-Balanced Learning: The training data is split into two families: the Preservation family (single-turn categorical/numerical tasks) and the Trajectory family (multi-round metric/temporal tasks). The authors use a balanced objective function to ensure that neither family dominates the shared model parameters, preventing the "lock-in" effect.

Additionally, the method retains Native-Format Targets. Rather than forcing all answers into a single canonical format, it keeps units, precision, and timestamps in their original source format to preserve task-specific semantics.

Figure 2: Overview of the ChronoReplay training stages.

At inference time, they introduce Adaptive Frame Selection, a policy that selects the most relevant historical frames based on the scope of the query (e.g., looking at the "dense tail" for current state vs. a "global uniform" sample for layout questions).

Datasets

The model was trained on a massive pool of spatial data:

  • Preservation Family: 33,180 sequences (30,635 categorical, 2,545 numerical).
  • Trajectory Family: 6,381 episodes (45,006 supervised turns).

Evaluation was performed on three major benchmarks:

  • OVO-S-Bench: Categorical reasoning on egocentric perception.
  • OST-Bench: Agent state, visible info, and spatial relations over multi-round exploration.
  • UCS-Bench: 8,114 timestamp-causal questions on trajectories and proximity.

Key Insights and Ideas

  • Chronology is a Feature: Replaying data in order isn't just a data organization choice; it's a supervision signal. The authors found that chronological replay alone provided a 2.41-point boost on OST-Bench compared to a shuffled control.
  • The "Lock-in" is Real: The paper empirically demonstrates that training a Qwen3-VL-8B model only on categorical data actually lowered its metric performance on OST-Bench from 50.52 to 47.48.
  • Native Formats Matter: Converting all spatial answers to a single "canonical" JSON format redistributed gains but didn't improve everything uniformly. Keeping native formats (like meters or timestamps) helps preserve the specific distinctions required by different tasks.
  • Inference-Time Efficiency: Adaptive frame selection allows the model to achieve higher accuracy (+4.09 points on OVO-S) while using fewer frames on average (80.5 vs 128), reducing computational cost.

Key Results

ChronoReplay shows significant across-the-board improvements using a single 8B parameter checkpoint:

  • OST-Bench: A massive +13.41 point gain over the base model, with the largest improvements in "Direction Estimation" (+26.86) and "Order Judgement" (+20.57).
  • UCS-Bench: A +7.44 point improvement, showing positive transfer in all eight subcategories.
  • OVO-S-Bench: An overall gain of +4.91 points, making the 8B model numerically comparable to much larger published references like the 235B Qwen3-VL variant.
Table 1: Complete task-family results on OVO-S-Bench (%), twelve official task families. Our rows report accuracy within each family; Overall is their unweighted mean. Published reference scores are quoted from the benchmark paper. Uniform-128 is the matched training comparison; the final row applies adaptive access to the same trained model.
Table 2: Fine-grained results on OST-Bench (%), official fifteen subtypes. Published rows are reference points under their reported full-split protocols; shaded rows use our matched scene-disjoint evaluation. ∆is ChronoReplay minus base. JUD., TEMP., CNT., and EST. denote judgement, temporal localization, counting, and estimation.

Limitations and Future Work

  • Environment Scope: The evaluation was primarily focused on indoor scenes (ScanNet, Matterport3D). Performance in complex outdoor or highly dynamic environments remains untested.
  • Memory Architecture: ChronoReplay is a training recipe, not a new architecture. While it improves how the model uses its internal state, it does not add an explicit long-term memory module, which might be necessary for hour-long video streams.
  • Mechanism Interpretability: While the authors identify "interface lock-in" as an empirical pattern, the exact internal representation changes that cause or solve it are not yet fully understood.

Future work could explore how these balanced training objectives interact with explicit memory architectures like scene graphs or voxel maps.

Summary updated

Abstract

Training a spatial VLM to answer which side an object lies on does not require it to preserve the metric displacement that determines that answer, while supervision for a distance estimate may leave categorical relations largely unconstrained. When post-training emphasizes one readout, improvements can therefore concentrate on that interface while transfer to others remains weak or even degrades; we call this observed specialization interface lock-in. ChronoReplay addresses it by replaying heterogeneous spatial supervision as chronologically ordered episodes, retaining source-native targets, and balancing supervision across families. Under the same observation access, a single ChronoReplay checkpoint improves OVO-S-Bench, OST-Bench, and UCS-Bench together, including a 13.41-point gain on OST-Bench, and achieves the highest three-benchmark mean among the supervision settings compared here. Same-data controls further separate the ingredients of this transfer: answer serialization redistributes gains across OST and UCS, family-balanced learning improves over ordinary joint SFT and natural turn weighting, and chronological replay exceeds a shifted-order control by about 2.41 OST points across two paired seeds. Adaptive frame selection then changes evidence access while keeping the trained model fixed, providing an additional inference-time gain at lower visual cost. Streaming spatial performance depends on what distinctions supervision preserves, how heterogeneous histories are organized during learning, and what causal evidence inference makes available at test time.
BibTeX
@misc{hu2026chronoreplay,
  author = {Yihao Hu and Yitong Lin and Pan Wang and Zhihao Wen and Hang Wang and Yiwen Zhao and Deyuan Liu and Yutong Li and Mingda Wan and Yuchen Mao and Tao Lin},
  title = {{ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence}},
  year = {2026},
  eprint = {2610.34283},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34283},
}
Figure