TL;DR
ChronoReplay is a novel post-training recipe that enables a single Vision-Language Model (VLM) to master diverse spatial tasks—categorical, metric, and temporal—simultaneously. By replaying heterogeneous spatial data in its original chronological order and balancing different supervision "families," it overcomes "interface lock-in" to achieve state-of-the-art results across multiple streaming spatial benchmarks.
What Problem Does This Paper Address?
Current spatial VLMs suffer from a phenomenon the authors call interface lock-in. When you train a model to answer categorical questions (e.g., "Is the chair on the left?"), it often loses the ability to perform metric estimation (e.g., "How many meters away is the chair?"), and vice versa. Improvements in one "interface" often come at the cost of degradation in others.
Furthermore, spatial intelligence in a streaming context (like a robot moving through a house) is inherently causal. An object's position relative to the observer depends entirely on the history of the observer's motion. If a model is trained on isolated snapshots or shuffled data, it loses the "temporal origin" that gives spatial coordinates their meaning.
Methods
The authors propose ChronoReplay, which consists of two primary components:
- Causal Replay Construction: Instead of treating training data as independent question-answer pairs, ChronoReplay organizes data into chronologically ordered episodes. Each target answer is attached to its "causal prefix"—the exact sequence of observations leading up to that moment. This forces the model to learn how spatial state accumulates over time.
- Family-Balanced Learning: The training data is split into two families: the Preservation family (single-turn categorical/numerical tasks) and the Trajectory family (multi-round metric/temporal tasks). The authors use a balanced objective function to ensure that neither family dominates the shared model parameters, preventing the "lock-in" effect.
Additionally, the method retains Native-Format Targets. Rather than forcing all answers into a single canonical format, it keeps units, precision, and timestamps in their original source format to preserve task-specific semantics.
At inference time, they introduce Adaptive Frame Selection, a policy that selects the most relevant historical frames based on the scope of the query (e.g., looking at the "dense tail" for current state vs. a "global uniform" sample for layout questions).
Datasets
The model was trained on a massive pool of spatial data:
- Preservation Family: 33,180 sequences (30,635 categorical, 2,545 numerical).
- Trajectory Family: 6,381 episodes (45,006 supervised turns).
Evaluation was performed on three major benchmarks:
- OVO-S-Bench: Categorical reasoning on egocentric perception.
- OST-Bench: Agent state, visible info, and spatial relations over multi-round exploration.
- UCS-Bench: 8,114 timestamp-causal questions on trajectories and proximity.
Key Insights and Ideas
- Chronology is a Feature: Replaying data in order isn't just a data organization choice; it's a supervision signal. The authors found that chronological replay alone provided a 2.41-point boost on OST-Bench compared to a shuffled control.
- The "Lock-in" is Real: The paper empirically demonstrates that training a Qwen3-VL-8B model only on categorical data actually lowered its metric performance on OST-Bench from 50.52 to 47.48.
- Native Formats Matter: Converting all spatial answers to a single "canonical" JSON format redistributed gains but didn't improve everything uniformly. Keeping native formats (like meters or timestamps) helps preserve the specific distinctions required by different tasks.
- Inference-Time Efficiency: Adaptive frame selection allows the model to achieve higher accuracy (+4.09 points on OVO-S) while using fewer frames on average (80.5 vs 128), reducing computational cost.
Key Results
ChronoReplay shows significant across-the-board improvements using a single 8B parameter checkpoint:
- OST-Bench: A massive +13.41 point gain over the base model, with the largest improvements in "Direction Estimation" (+26.86) and "Order Judgement" (+20.57).
- UCS-Bench: A +7.44 point improvement, showing positive transfer in all eight subcategories.
- OVO-S-Bench: An overall gain of +4.91 points, making the 8B model numerically comparable to much larger published references like the 235B Qwen3-VL variant.
Limitations and Future Work
- Environment Scope: The evaluation was primarily focused on indoor scenes (ScanNet, Matterport3D). Performance in complex outdoor or highly dynamic environments remains untested.
- Memory Architecture: ChronoReplay is a training recipe, not a new architecture. While it improves how the model uses its internal state, it does not add an explicit long-term memory module, which might be necessary for hour-long video streams.
- Mechanism Interpretability: While the authors identify "interface lock-in" as an empirical pattern, the exact internal representation changes that cause or solve it are not yet fully understood.
Future work could explore how these balanced training objectives interact with explicit memory architectures like scene graphs or voxel maps.
Summary updated