ProV-Mem: Proactive Memory for Streaming Video Question Answering

Bingbing Wang, Jing Li, Han Gao, Yishi Lin, Ruifeng Xu, Peng HearXiv 2026 · arXiv

TL;DR

ProV-Mem is a training-free framework that enables Vision-Language Models (VLMs) to autonomously engage with streaming video by managing "when," "what," and "whether" to speak through a three-tiered proactive memory system. It significantly outperforms existing methods on the ProactiveVideoQA benchmark while introducing a more rigorous, cost-aware evaluation metric (CAUC) to prevent "reward hacking" through repetitive or verbose responses.

What Problem Does This Paper Address?

Traditional Video Question Answering (VQA) is a passive affair: a user provides a video and a query, and the model responds. However, real-world applications like live moderation or embodied AI agents require Proactive-VQA. In this setting, an agent must continuously monitor an unbounded video stream and autonomously decide on the timing, content, and necessity of its responses without explicit prompts.

Existing streaming systems face a "dual predicament":

  1. Passive Memory: Most architectures use lightweight feature caches that store context but don't actively participate in decision-making. This leads to "leaky" history where models miss critical events or fall into repetitive, templated loops.
  2. Leaky Metrics: The standard metric, Proactive Area Under the Curve (PAUC), can be "gamed." Models can inflate their scores by firing frequently on a single event, accumulating window duration rather than providing new information.
Figure 1: Comparison of (a) existing methods and (b) our ProV-Mem method.

Methods

The authors propose ProV-Mem, a training-free framework built around a frozen VLM backbone (like Qwen2.5-VL). Instead of a monolithic "head" trying to learn when to speak, ProV-Mem factorizes the decision into three lightweight, non-parametric memory tiers:

  • Temporal-Trigger Memory (When): Uses deterministic gating to query the backbone at regular intervals (stride $K$). It includes a "silence-recovery" mechanism to force a query if the model has been dormant for too long, ensuring it doesn't miss sparse events.
  • Instructional Memory (What): Employs a fixed prompting prior that enforces a "canonical surface form." This guides the VLM to produce concise, visually grounded, and objective descriptions, making the output easier to filter for redundancy.
  • Episodic Memory (Whether): Acts as an emission gate. It caches the most recent response and compares it to new candidates using character-level similarity (Longest Common Subsequence). If a new response is too similar to the last one, it's suppressed unless a significant "cooldown" period has passed.
Figure 2: Framework of our ProV-Mem method.

To address the evaluation loophole, the authors introduce Cost-Aware Area Under the Curve (CAUC). This metric explicitly penalizes three "costs": redundancy (similarity to history), excessive frequency (chatter), and token inefficiency (unnecessary verbosity).

Datasets

The framework was evaluated on the ProactiveVideoQA benchmark, which consists of four distinct streaming scenarios:

  • WEB: Continuous, low-latency web video narration.
  • EGO: High-frequency first-person activity tracking (dense, repetitive steps).
  • TV: Subtitle-centric dialogue QA requiring alignment of visual and textual evidence.
  • VAD: Video Anomaly Detection, characterized by highly sparse but critical events.

Key Insights and Ideas

  • Proactive vs. Passive: The core "aha" moment is that memory shouldn't just be a storage bin; it should be a controller. By giving each memory tier a specific structural role in the decision chain, the system avoids the representational collapse seen in models that try to compress all history into a single hidden state.
  • Training-Free Robustness: Surprisingly, this non-parametric approach outperforms RL-trained trigger policies. The authors argue that RL models often overfit to the temporal statistics of their training data, causing them to fail when event density shifts in new domains.
  • Monotonic Scaling: Unlike monolithic designs where increased backbone capacity can lead to interference between temporal reasoning and semantic generation, ProV-Mem scales monotonically. As the VLM gets bigger, performance consistently improves because the memory management is handled externally.
Figure 3: Backbone scaling on ProactiveVideoQA with the ProV-Mem.

Key Results

  • State-of-the-Art Performance: ProV-Mem achieved the best CAUC across all four scenarios and the best PAUC on WEB, EGO, and TV. For instance, on the EGO task, it reached a PAUC of 70.87, dwarfing the previous best online streaming model, MMDuet2 (RL), which scored 31.58.
  • Efficiency: ProV-Mem maintains a remarkably low per-response inference latency of 1.30 seconds, significantly faster than offline closed-source models like GPT-4o-mini (~21s) or Gemini-2.5-Pro (~20s) adapted for streaming.
  • Metric Integrity: The gap between PAUC and CAUC scores for baselines confirmed that many existing models were indeed "reward hacking" by over-firing. ProV-Mem's lead on both metrics suggests its gains come from genuine content management.

Limitations and Future Work

The primary exception to ProV-Mem's dominance was the PAUC score in the VAD (Anomaly Detection) scenario. The authors note that detecting rare anomalies may require stronger parametric event priors—essentially, a "gut feeling" for what constitutes an unusual event—that surface-level lexical memory cannot yet provide.

Future work may involve extending the temporal memory tier to incorporate more sophisticated, perhaps learned, event-scouting mechanisms while maintaining the lightweight, non-parametric advantages of the overall framework.

Summary updated

Abstract

Proactive streaming video question answering requires a model to continuously monitor an incoming stream and autonomously decide when to speak, what to say, and whether to emit a response. Existing streaming systems face a fundamental difficulty: retaining long visual histories is costly, whereas discarding them can lead to missed events and repeated responses. We propose \textbf{ProV-Mem}, a training-free proactive memory framework built around a frozen VLM backbone. ProV-Mem decomposes the proactive decision into three lightweight memory tiers: a temporal-trigger memory controls when to query the backbone and recovers from prolonged silence, an instructional memory guides what to say through a fixed prompting prior for concise and visually grounded generation, and an episodic memory determines whether to emit by filtering near-duplicate responses. Under an idealized conditional-sufficiency assumption, we derive a stagewise approximation bound relating the decomposed process to a full-state joint policy. We further introduce Cost-Aware Area Under the Curve (\textbf{CAUC}), which extends PAUC by explicitly discounting redundant, overly frequent, and unnecessarily long emissions. Experiments on the four scenarios of ProactiveVideoQA show that ProV-Mem achieves the best CAUC across all scenarios and the best PAUC on WEB, EGO, and TV, while maintaining low per-response inference latency.
BibTeX
@misc{wang2026prov,
  author = {Bingbing Wang and Jing Li and Han Gao and Yishi Lin and Ruifeng Xu and Peng He},
  title = {{ProV-Mem: Proactive Memory for Streaming Video Question Answering}},
  year = {2026},
  eprint = {2610.23998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23998},
}
Figure