BLIP-Live: Balancing Timeliness & Firing Precision in Proactive Streaming Video Agents

Kanchana Ranasinghe, Honglu Zhou, Dominick Reilly, Le Xue, Shusheng Yang, Juan Carlos Niebles, Silvio Savarese, Ran XuarXiv 2026 · arXiv

TL;DR

BLIP-Live is a unified framework for building proactive streaming video agents that know not just what to say, but exactly when to say it—and when to stay silent. By introducing a massive human-verified dataset (780K responses), a new metric (PAUC-F) that penalizes "spammy" behavior, and an optimized training pipeline, the authors achieve state-of-the-art performance, outperforming previous baselines by up to 44%.

What Problem Does This Paper Address?

Most current video-language models (VLMs) operate in an "offline" mode: they ingest a complete video clip and then answer a query. However, real-world interactive agents—like a robotic assistant or a smart home monitor—must operate in a "streaming" fashion. They observe video frame-by-frame and must decide in real-time when sufficient evidence exists to trigger a response.

The core challenge is balancing timeliness with firing precision. If an agent responds too late, the information is useless; if it responds too early or repeats itself constantly (a behavior called "over-firing"), it becomes intrusive and annoying. The authors identify a critical gap: existing evaluation metrics often reward models for "spamming" responses to ensure they hit a target window, failing to penalize the lack of silence.

Methods

The BLIP-Live framework unifies data, metrics, and training around the concept of "effective causal response decisions."

  1. PAUC-F Metric: The authors propose a new metric that combines the existing PAUC (which measures quality-weighted timeliness) with a Precision term. This ensures models are rewarded for being fast and for not firing unnecessarily.
  2. Inference Harness: To enable "frozen" offline VLMs to work in streaming settings, the authors developed a model-agnostic harness. As shown in Figure 3, it manages a rolling window of past frames and a "textual memory" of previous interactions to maintain context without seeing the future. Figure 3: Inference Harness. At each streaming timestep t, the harness constructs an input from a system prompt, a rolling past-frame window, and memory of user queries and prior responses.

  3. Streaming Model Training: For models that are fine-tuned, the authors use a three-state formulation:

  4. Silence (S): The agent should remain quiet.
  5. Response (R): The exact moment the agent should provide an answer.
  6. Hold (H): The event is still ongoing, so the agent should stay silent to avoid redundancy. This "Efficient Causal SFT" (Supervised Fine-Tuning) pipeline is illustrated in Figure 4. Figure 4: Streaming Model Training. The VLM is supervised on state targets (silence, response, and hold) and the accompanying response text.
Figure 3: Inference Harness. At each streaming timestep t, the harness constructs an input from a system prompt (task and output schema), a rolling past-frame window, and memory of user queries and prior responses. A frozen offline VLM is queried at the target frame rate; structured output is parsed into either silence or a response at t, and accepted responses are updated into memory.
Figure 4: Streaming Model Training. Each training sequence concatenates a system prompt, memory of user queries / past responses, and an interleaved frame-target stream. Prompt, memory, and visual tokens receive no loss; the VLM is supervised on state targets (silence, response, and hold) and the accompanying response text. One video is split into multiple chunks so that long streams remain within the training context.

Datasets

The paper introduces BLIP-Live-780K, a large-scale dataset containing 784,944 proactive responses across nearly 178,000 videos. Unlike previous datasets that relied purely on automated generation, BLIP-Live-780K underwent a rigorous three-stage pipeline including automated VLM generation followed by comprehensive human verification and expert quality review. This ensures that the "silence" intervals are as accurately annotated as the "response" moments.

For evaluation, the authors test across four major benchmarks: ProactiveVideoQA, ESTP-Bench, StreamingBench, and OVOBench.

Key Insights and Ideas

  • The "Hold" State is a Game Changer: One of the most significant "aha" moments is the importance of the Hold state. By explicitly training the model to recognize when an event is continuing but doesn't require a new response, they drastically reduced redundant firing.
  • Metrics Drive Behavior: The authors demonstrate that models optimized for older metrics like PAUC often exhibit "spammy" behavior because those metrics don't penalize over-firing. Figure 2 shows how PAUC-F correctly identifies and penalizes these undesired patterns while older metrics fail. Figure 2: PAUC-F vs Prior Metrics. PAUC-F correctly penalizes undesired "spam" response patterns that older metrics (PAUC, ESTP-F1) incorrectly reward.

  • Harness Tuning vs. Training: Interestingly, the authors found that even without retraining a model, carefully tuning the "inference harness" (the system prompt and memory management) using an agent-assisted search can significantly improve proactivity in off-the-shelf VLMs.

Figure 2: PAUC-F vs Prior Metrics: We simulate excessive firing be- haviour (undesired) across WEB & EGO splits of ProactiveVideoQA (Wang et al., 2025b) and Explicit Proactive subset of ESTP-Bench (Yu et al., 2025) benchmarks, reporting our proposed PAUC-F metric along- side PAUC & ESTP-F1 metrics. Along the x-axes, we repeat ground- truth responses (Oracle) 5 additional times within golden windows (Or- acle + 5x) and every 0.5s throughout the video (Spam 0.5). PAUC-F correctly identified these undesired response patterns while PAUC and ESTP-F1 incorrectly rewards all or some of these undesired test cases.

Key Results

  • State-of-the-Art Performance: The BLIP-Live-Qwen model achieved top results across all four proactive streaming benchmarks.
  • Massive Gains: On the ProactiveVideoQA benchmark, BLIP-Live-Qwen outperformed the previous state-of-the-art (MMDuet2) by 44% in relative performance.
  • Beating Proprietary Models: In several settings, the open-weight BLIP-Live models outperformed much larger closed-source API models, including versions of GPT-4 and Gemini, particularly in their ability to maintain firing precision.
  • Human Preference: In a blind study, human raters preferred the outputs of the PAUC-F optimized agents over 90% of the time, citing better timing and less repetition.

Limitations and Future Work

  • Temporal Resolution: The current system operates at a decision rate of 1 Hz (one decision per second). While standard for current research, real-world applications might require higher frequency (e.g., 10-30 Hz) for smoother interaction.
  • Computational Overhead: Maintaining a long-term "textual memory" and processing rolling windows of frames can be computationally expensive as video length increases.
  • Future Directions: The authors suggest exploring more complex multi-turn proactive dialogues and improving the model's ability to handle overlapping events in highly dynamic scenes.

Summary updated

Abstract

Streaming video understanding is becoming increasingly important for real-world interactive systems. Such systems must be proactive for natural and useful human interaction: they should respond as soon as sufficient evidence becomes available while avoiding premature, redundant, or incorrect responses. We present BLIP-Live, a framework unifying data, metrics, harness, and training around a shared formulation of proactivity as effective causal response decisions balancing the two competing objectives of responding vs silence. First, we construct BLIP-Live-780K dataset containing human annotations of proactive response content and timing, which defines this desired behavior. We then propose PAUC-F, a metric to effectively measure our desired proactive behavior. PAUC-F combines existing response timeliness metrics with firing precision to reward early useful responses without rewarding excessive firing. With our desired behavior defined and measurable, we finally investigate model-agnostic inference harnesses that draw out proactive streaming capabilities of offline vision language models (VLMs), together with training pipelines that learn to further improve their proactive responsiveness. Across both settings, we find that combinations of sliding visual windows, online interaction memory, and structured response outputs consistently yield strongest proactive performance. Our experiments result in two models, with our BLIP-Live-Qwen model achieving state-of-the-art performance across four proactive streaming benchmarks, with gains up to 44% over baselines and outperforming multiple closed API models. Our code and data will be released publicly.
BibTeX
@misc{ranasinghe2026blip,
  author = {Kanchana Ranasinghe and Honglu Zhou and Dominick Reilly and Le Xue and Shusheng Yang and Juan Carlos Niebles and Silvio Savarese and Ran Xu},
  title = {{BLIP-Live: Balancing Timeliness \& Firing Precision in Proactive Streaming Video Agents}},
  year = {2026},
  eprint = {2611.21553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21553},
}
Figure