TL;DR
ProV-Mem is a training-free framework that enables Vision-Language Models (VLMs) to autonomously engage with streaming video by managing "when," "what," and "whether" to speak through a three-tiered proactive memory system. It significantly outperforms existing methods on the ProactiveVideoQA benchmark while introducing a more rigorous, cost-aware evaluation metric (CAUC) to prevent "reward hacking" through repetitive or verbose responses.
What Problem Does This Paper Address?
Traditional Video Question Answering (VQA) is a passive affair: a user provides a video and a query, and the model responds. However, real-world applications like live moderation or embodied AI agents require Proactive-VQA. In this setting, an agent must continuously monitor an unbounded video stream and autonomously decide on the timing, content, and necessity of its responses without explicit prompts.
Existing streaming systems face a "dual predicament":
- Passive Memory: Most architectures use lightweight feature caches that store context but don't actively participate in decision-making. This leads to "leaky" history where models miss critical events or fall into repetitive, templated loops.
- Leaky Metrics: The standard metric, Proactive Area Under the Curve (PAUC), can be "gamed." Models can inflate their scores by firing frequently on a single event, accumulating window duration rather than providing new information.
Methods
The authors propose ProV-Mem, a training-free framework built around a frozen VLM backbone (like Qwen2.5-VL). Instead of a monolithic "head" trying to learn when to speak, ProV-Mem factorizes the decision into three lightweight, non-parametric memory tiers:
- Temporal-Trigger Memory (When): Uses deterministic gating to query the backbone at regular intervals (stride $K$). It includes a "silence-recovery" mechanism to force a query if the model has been dormant for too long, ensuring it doesn't miss sparse events.
- Instructional Memory (What): Employs a fixed prompting prior that enforces a "canonical surface form." This guides the VLM to produce concise, visually grounded, and objective descriptions, making the output easier to filter for redundancy.
- Episodic Memory (Whether): Acts as an emission gate. It caches the most recent response and compares it to new candidates using character-level similarity (Longest Common Subsequence). If a new response is too similar to the last one, it's suppressed unless a significant "cooldown" period has passed.
To address the evaluation loophole, the authors introduce Cost-Aware Area Under the Curve (CAUC). This metric explicitly penalizes three "costs": redundancy (similarity to history), excessive frequency (chatter), and token inefficiency (unnecessary verbosity).
Datasets
The framework was evaluated on the ProactiveVideoQA benchmark, which consists of four distinct streaming scenarios:
- WEB: Continuous, low-latency web video narration.
- EGO: High-frequency first-person activity tracking (dense, repetitive steps).
- TV: Subtitle-centric dialogue QA requiring alignment of visual and textual evidence.
- VAD: Video Anomaly Detection, characterized by highly sparse but critical events.
Key Insights and Ideas
- Proactive vs. Passive: The core "aha" moment is that memory shouldn't just be a storage bin; it should be a controller. By giving each memory tier a specific structural role in the decision chain, the system avoids the representational collapse seen in models that try to compress all history into a single hidden state.
- Training-Free Robustness: Surprisingly, this non-parametric approach outperforms RL-trained trigger policies. The authors argue that RL models often overfit to the temporal statistics of their training data, causing them to fail when event density shifts in new domains.
- Monotonic Scaling: Unlike monolithic designs where increased backbone capacity can lead to interference between temporal reasoning and semantic generation, ProV-Mem scales monotonically. As the VLM gets bigger, performance consistently improves because the memory management is handled externally.
Key Results
- State-of-the-Art Performance: ProV-Mem achieved the best CAUC across all four scenarios and the best PAUC on WEB, EGO, and TV. For instance, on the EGO task, it reached a PAUC of 70.87, dwarfing the previous best online streaming model, MMDuet2 (RL), which scored 31.58.
- Efficiency: ProV-Mem maintains a remarkably low per-response inference latency of 1.30 seconds, significantly faster than offline closed-source models like GPT-4o-mini (~21s) or Gemini-2.5-Pro (~20s) adapted for streaming.
- Metric Integrity: The gap between PAUC and CAUC scores for baselines confirmed that many existing models were indeed "reward hacking" by over-firing. ProV-Mem's lead on both metrics suggests its gains come from genuine content management.
Limitations and Future Work
The primary exception to ProV-Mem's dominance was the PAUC score in the VAD (Anomaly Detection) scenario. The authors note that detecting rare anomalies may require stronger parametric event priors—essentially, a "gut feeling" for what constitutes an unusual event—that surface-level lexical memory cannot yet provide.
Future work may involve extending the temporal memory tier to incorporate more sophisticated, perhaps learned, event-scouting mechanisms while maintaining the lightweight, non-parametric advantages of the overall framework.
Summary updated