StreamReady: Learning What to Answer and When in Long Streaming Videos

Shehreen Azad, Vibhav Vineet, Yogesh S Rawat

TL;DR

StreamReady introduces a "readiness-aware" framework for streaming video understanding, ensuring models answer not just correctly, but precisely when the supporting visual evidence appears. By combining a hierarchical visual memory tree with a learnable readiness token, it avoids premature speculation and unnecessary delays, outperforming existing models across multiple streaming and offline benchmarks.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at short video clips, processing long, continuous video streams remains a challenge. Existing streaming video frameworks focus almost entirely on what to answer (correctness), assuming the evidence is already present in the past video history.

However, real-world applications like robotics, surveillance, and assistive systems often require proactive (future-dependent) reasoning, where a question is posed before the evidence occurs. Current models lack a mechanism to decide when they have observed enough information to respond. As a result, they either speculate blindly too early or introduce significant real-time delays by waiting too long.

Methods

The authors formalize this challenge as readiness-aware streaming video understanding and introduce three core components:

  1. Answer Readiness Score (ARS): A new timing-aware evaluation metric that applies asymmetric penalties. It heavily penalizes early answers (to discourage hallucination/speculation) and mildly penalizes late answers (to tolerate slight hesitation). Combined with raw accuracy, it defines "effective accuracy."
  2. Hierarchical Memory Storage: To handle long video streams without memory explosion, StreamReady uses a Visual Memory Tree ($M_V$) that abstracts frames into three levels: short-term raw frames (a FIFO buffer), mid-level centroids, and coarse prototypes generated via K-means and Exponential Moving Average (EMA) clustering. A Contextual Memory Bank ($M_C$) stores past question-answer pairs to maintain multi-turn context.
  3. Readiness Mechanism: Instead of relying on a heavy auxiliary model, the framework appends a learnable <RDY> token to the long-term reasoning features within a dual-branch Q-Former. A lightweight Readiness Head monitors this token and outputs a score ($R_{pred}$). The model only triggers the LLM to answer when this score crosses a predefined threshold.

To train the readiness head without requiring expensive ground-truth timestamp annotations, the authors use weak pseudo-supervision. They calculate the similarity between the current reasoning state and the mid-level visual memory centroids to automatically define pseudo-positive and pseudo-negative temporal regions.

Figure 2. Framework Overview. StreamReady encodes stream- ing videos into a visual memory tree and reasons through short and long-term branches. A learnable <RDY> token, guided by a readi- ness head, gates the reasoning output until sufficient evidence is observed. Once ready, the long-term representation, enriched with contextual information from past QA pairs, is sent to the LLM for answering, enabling readiness-aware streaming behavior.

Datasets

The authors introduce ProReady-QA, a new benchmark specifically designed for proactive streaming QA. It features:

  • 10 one-hour Ego-4D videos and 22 half-hour MovieNet videos.
  • 5,000 proactive multi-turn question-answer pairs.
  • Human-refined, explicitly annotated answer evidence windows.
  • Five distinct tasks: Sequential Steps Recognition (SSR), Repetitive Event Counting (REC), Clues Reveal Responding (CRR), Causal Trigger Detection (CTD), and Goal-State Detection (GSD).

For broader validation, the model was also evaluated on four existing streaming benchmarks (StreamingBench, OVOBench, OVBench, VStream-QA) and four offline long-video benchmarks (VideoMME, MLVU, MVBench, EgoSchema).

Key Insights and Ideas

  • Coupling Timing with Reasoning: A key architectural "aha moment" is embedding the <RDY> token directly inside the long-term reasoning module. This allows the timing decision to co-evolve with the query-aligned visual evidence, making the readiness detection highly reliable compared to detached heuristic triggers.
  • The Illusion of Raw Accuracy: Standard accuracy metrics fail to capture timing errors. As illustrated in the paper, two models can achieve identical raw accuracy, but one might be guessing blindly before the event happens. ARS exposes these hidden drops in effective accuracy.
Figure 1. Readiness-aware streaming video understanding. Left: In proactive streaming settings, questions can precede their supporting evidence, requiring the model to monitor the evolving video and answer once the relevant cues appear. Right: Under our readiness-aware formulation, effective accuracy jointly reflects answer correctness and timing via the Answer Readiness Score (ARS). Although all models achieve similar raw accuracy on this example, ARS reveals sharp performance drops for early (hallucinatory) or late (delayed) answers. In contrast, StreamReady responds within the evidence window, preserving high effective accuracy by answering at the appropriate moment.
  • Fixed-Size Memory Scales Best: By compressing older frames into a fixed number of centroids and prototypes, StreamReady maintains flat, stable latency and memory usage even as the video length scales up to tens of thousands of frames.

Key Results

  • ProReady-QA Performance: StreamReady achieved the highest raw accuracy (56.4%) and ARS (0.69) across all tasks, outperforming the best online baseline (StreamBridge) by ~3% in accuracy and ~9% in ARS.
  • Generalization: Even on benchmarks where timing is not explicitly evaluated, StreamReady consistently outperformed prior state-of-the-art methods (e.g., achieving up to ~5% gains on proactive tasks in StreamingBench). This indicates that its hierarchical memory retrieval inherently improves long-horizon video reasoning.
  • Computational Efficiency: While baseline models like Qwen-2-VL suffered from out-of-memory failures on long streams due to full-attention accumulation, StreamReady maintained constant memory overhead and stable inference latency.

Limitations and Future Work

The authors note that the readiness mechanism relies on visual similarity to generate its weak pseudo-supervision signals. This approach might struggle in highly static scenes or visually repetitive environments where distinct events look identical, potentially leading to unstable readiness scores.

Future work could focus on incorporating finer semantic or textual cues into the pseudo-supervision process to handle visually ambiguous streams more robustly.

Summary updated

Abstract

Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the **Answer Readiness Score (ARS)**, a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce **StreamReady**, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce **ProReady-QA**, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.
BibTeX
@inproceedings{azad2026streamready,
  author = {Shehreen Azad and Vibhav Vineet and Yogesh S Rawat},
  title = {{StreamReady: Learning What to Answer and When in Long Streaming Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=kwy3AAbWlw},
}
Figure