TL;DR
SimplePro is a training-free baseline for proactive video understanding that enables frozen Vision-Language Models (VLMs) to decide when to speak and what to say as a video unfolds. By combining structured streaming context with explicit reporting rules, it outperforms complex, reinforcement-learning-based systems on major benchmarks without any parameter updates.
What Problem Does This Paper Address?
Most modern Vision-Language Models (VLMs) excel at "offline" video understanding—you give them a complete video file, ask a question, and they provide an answer. However, real-world assistants (like a robot monitoring a kitchen or a security system) operate in a "proactive streaming" mode. They must watch a live feed and decide in real-time: Does what I just saw warrant a response, or should I stay silent?
This is surprisingly difficult. A proactive assistant faces three main pitfalls:
- Answering too early: Reporting an event before the evidence is sufficient (e.g., saying "the box is sealed" when the person has just started taping it).
- Answering too late: Missing the moment entirely.
- Redundancy: Repeating the same report over and over while an event is ongoing.
Previous research attempted to solve this by adding "trigger" modules or using task-specific Reinforcement Learning (RL) to teach models when to speak. While effective, these methods are expensive to train, require specialized datasets, and must be redone every time a better VLM backbone is released.
Methods
The authors introduce SimplePro, a "training-free" approach. Instead of changing the model's weights, they change how the model's "working memory" and instructions are structured.
At each decision step, SimplePro feeds a frozen VLM three types of information:
- Recent Frames ($F_t$): High-resolution images of the most recent activity to provide clear evidence.
- Historical Observations ($H_t$): A pruned history of past frames and model responses to provide context for event changes.
- Answer Record ($A_t$): A dedicated log of what the model has already told the user. This is kept even if the corresponding images are pruned from history to save memory.
The core of the system is a set of Explicit Reporting Rules provided in the prompt. These rules instruct the model to:
- Specify Requirements: Distinguish between single-target questions (answer once) and recurring events (answer every time).
- Avoid Repetition: Explicitly check the Answer Record to ensure it isn't reporting a continuation of a previously reported event.
- Allow Corrections: Revise previous answers if new, better evidence arrives.
Finally, a simple Duplicate-Response Filter catches any consecutive identical answers before they reach the user, though the raw outputs are kept in the internal history for the model's own reference.
Datasets
The authors evaluated SimplePro across three diverse benchmarks that test different aspects of proactivity:
- StreamingBench-PO: Tests "first-response" timing—how accurately the model hits the start of an event.
- ProactiveVideoQA: Measures the quality of answers over the entire duration of a video using the PAUC (Partial Area Under the Curve) metric.
- OVO-Bench: Evaluates "continued event reporting," where a model must track and report recurring or multi-step events.
Key Insights and Ideas
- Frozen VLMs are "Proactive-Ready": The most significant finding is that general-purpose VLMs (like Qwen2-VL) already possess the reasoning capabilities for proactive tasks. They just need the right "scaffolding" (context and rules) to apply them.
- The Answer Record is Vital: There is a decoupling between visual memory and conversational memory. The authors found that even if you prune old images to fit into a 32K token context window, you must keep the text of prior answers. Without this record, models frequently repeat themselves.
- History Compression is a Free Lunch: Reducing the resolution of historical frames (while keeping recent frames high-res) cuts input tokens by over 36% with almost no impact on accuracy. This suggests that for history, the model needs "semantic" context more than fine-grained detail.
- Rules > RL: SimplePro's rule-based prompting on a 4B parameter model outperformed specialized systems that underwent extensive reinforcement learning.
Key Results
SimplePro sets a new state-of-the-art for training-free proactive systems:
- StreamingBench-PO: Achieved 54.80% Acc@2s, outperforming the previous best (SyncPro-RL) by 13.6 percentage points.
- ProactiveVideoQA: Reached a mean PAUC of 52.18, a 12.4-point lead over MMDuet2.
- OVO-Bench: Reached an F1 score of 35.23, slightly edging out specialized RL-trained models.
- Scaling: Performance consistently improved as the backbone size increased from 2B to 32B parameters, proving the method's robustness across model scales.
- Efficiency: Despite using a larger backbone (4B vs 3B), SimplePro maintained a decision latency of ~526ms, comparable to existing specialized streaming models.
Limitations and Future Work
While SimplePro is highly effective, the authors note a few areas for growth:
- Context Window Constraints: Even with pruning, very long streaming sessions will eventually hit the token limit of the underlying VLM.
- Prompt Sensitivity: The system relies on the VLM's ability to follow complex instructions. While modern 4B+ models handle this well, smaller or less capable models might struggle.
- Future Directions: The authors suggest exploring more sophisticated memory retrieval (beyond simple pruning) and testing the pipeline on even more complex, multi-modal streaming environments.
Summary updated