TL;DR
StreamReady introduces a "readiness-aware" framework for streaming video understanding, ensuring models answer not just correctly, but precisely when the supporting visual evidence appears. By combining a hierarchical visual memory tree with a learnable readiness token, it avoids premature speculation and unnecessary delays, outperforming existing models across multiple streaming and offline benchmarks.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MLLMs) excel at short video clips, processing long, continuous video streams remains a challenge. Existing streaming video frameworks focus almost entirely on what to answer (correctness), assuming the evidence is already present in the past video history.
However, real-world applications like robotics, surveillance, and assistive systems often require proactive (future-dependent) reasoning, where a question is posed before the evidence occurs. Current models lack a mechanism to decide when they have observed enough information to respond. As a result, they either speculate blindly too early or introduce significant real-time delays by waiting too long.
Methods
The authors formalize this challenge as readiness-aware streaming video understanding and introduce three core components:
- Answer Readiness Score (ARS): A new timing-aware evaluation metric that applies asymmetric penalties. It heavily penalizes early answers (to discourage hallucination/speculation) and mildly penalizes late answers (to tolerate slight hesitation). Combined with raw accuracy, it defines "effective accuracy."
- Hierarchical Memory Storage: To handle long video streams without memory explosion, StreamReady uses a Visual Memory Tree ($M_V$) that abstracts frames into three levels: short-term raw frames (a FIFO buffer), mid-level centroids, and coarse prototypes generated via K-means and Exponential Moving Average (EMA) clustering. A Contextual Memory Bank ($M_C$) stores past question-answer pairs to maintain multi-turn context.
- Readiness Mechanism: Instead of relying on a heavy auxiliary model, the framework appends a learnable
<RDY>token to the long-term reasoning features within a dual-branch Q-Former. A lightweight Readiness Head monitors this token and outputs a score ($R_{pred}$). The model only triggers the LLM to answer when this score crosses a predefined threshold.
To train the readiness head without requiring expensive ground-truth timestamp annotations, the authors use weak pseudo-supervision. They calculate the similarity between the current reasoning state and the mid-level visual memory centroids to automatically define pseudo-positive and pseudo-negative temporal regions.
Datasets
The authors introduce ProReady-QA, a new benchmark specifically designed for proactive streaming QA. It features:
- 10 one-hour Ego-4D videos and 22 half-hour MovieNet videos.
- 5,000 proactive multi-turn question-answer pairs.
- Human-refined, explicitly annotated answer evidence windows.
- Five distinct tasks: Sequential Steps Recognition (SSR), Repetitive Event Counting (REC), Clues Reveal Responding (CRR), Causal Trigger Detection (CTD), and Goal-State Detection (GSD).
For broader validation, the model was also evaluated on four existing streaming benchmarks (StreamingBench, OVOBench, OVBench, VStream-QA) and four offline long-video benchmarks (VideoMME, MLVU, MVBench, EgoSchema).
Key Insights and Ideas
- Coupling Timing with Reasoning: A key architectural "aha moment" is embedding the
<RDY>token directly inside the long-term reasoning module. This allows the timing decision to co-evolve with the query-aligned visual evidence, making the readiness detection highly reliable compared to detached heuristic triggers. - The Illusion of Raw Accuracy: Standard accuracy metrics fail to capture timing errors. As illustrated in the paper, two models can achieve identical raw accuracy, but one might be guessing blindly before the event happens. ARS exposes these hidden drops in effective accuracy.
- Fixed-Size Memory Scales Best: By compressing older frames into a fixed number of centroids and prototypes, StreamReady maintains flat, stable latency and memory usage even as the video length scales up to tens of thousands of frames.
Key Results
- ProReady-QA Performance: StreamReady achieved the highest raw accuracy (56.4%) and ARS (0.69) across all tasks, outperforming the best online baseline (StreamBridge) by ~3% in accuracy and ~9% in ARS.
- Generalization: Even on benchmarks where timing is not explicitly evaluated, StreamReady consistently outperformed prior state-of-the-art methods (e.g., achieving up to ~5% gains on proactive tasks in StreamingBench). This indicates that its hierarchical memory retrieval inherently improves long-horizon video reasoning.
- Computational Efficiency: While baseline models like Qwen-2-VL suffered from out-of-memory failures on long streams due to full-attention accumulation, StreamReady maintained constant memory overhead and stable inference latency.
Limitations and Future Work
The authors note that the readiness mechanism relies on visual similarity to generate its weak pseudo-supervision signals. This approach might struggle in highly static scenes or visually repetitive environments where distinct events look identical, potentially leading to unstable readiness scores.
Future work could focus on incorporating finer semantic or textual cues into the pseudo-supervision process to handle visually ambiguous streams more robustly.
Summary updated