TL;DR
StreamVibe introduces a framework and benchmark for proactive video agents that don't just wait for commands, but infer implicit and evolving user intents. By combining a streaming vision-language model (StreamVibe-VL) with a state-tracking "Harness," the system achieves a 10.7% improvement in response timing and quality over existing baselines.
What Problem Does This Paper Address?
Most current AI assistants are "passive"—they wait for an explicit request like "Set a timer" or "Notify me when the water boils." However, in complex real-world tasks (like cooking a new recipe or assembling furniture), a truly helpful agent needs to understand implicit intent (inferring help is needed because the user looks stuck) and dynamic intent (adapting when the user changes their mind or adds new constraints mid-task).
Existing models struggle with three main gaps:
- Reliance on explicit requests: They miss opportunities to help when the user hasn't asked a direct question.
- Insufficient state tracking: They often lose track of what has already been accomplished, leading to redundant or outdated advice.
- Limited adaptation: They fail to update their internal "to-do list" when user goals shift during an interaction.
Methods
The authors propose a two-part solution: a specialized model and an agentic framework.
1. StreamVibe-VL (The Model)
This is a streaming proactive model trained in two stages:
- Adaptive Streaming SFT: A supervised fine-tuning stage that teaches the model to balance "waiting" (observing) and "speaking" (responding). It uses weighted tokens to ensure the model doesn't become over-talkative or too silent.
- Localized Trajectory RL: Using Group Relative Policy Optimization (GRPO), the model is refined specifically around "help points." This avoids the "noise" of a full video and focuses the learning on critical moments where timing and content quality matter most.
2. StreamVibe-Harness (The Framework)
To handle long-term interactions, the authors developed a "Harness" that wraps around the model:
- Intent Lifecycle Management: It treats every user intent as a persistent state that can be active, satisfied, or revised.
- Hierarchical Memory: Instead of cramming the entire video history into the model's context window, the Harness uses a memory system that keeps recent observations fine-grained while progressively compressing older information.
Datasets
The researchers introduced StreamVibe-Bench, a comprehensive testbed for proactive interaction:
- Scale: 4.1K evaluation samples and 85K training samples.
- Diversity: Covers both physical (egocentric daily activities) and digital (GUI/computer use) environments.
- Intent Levels: It categorizes tasks into Explicit (monitoring a specific event), Implicit (inferring help from high-level goals), and Dynamic (handling revisions, compositions, or interruptions).
Key Insights and Ideas
- Proactivity is a Timing Problem: It’s not just about what to say, but when to say it. The paper shows that optimizing for "help points" via RL is much more effective than standard video-captioning approaches.
- State Tracking vs. Policy: The authors found that a model's "policy" (the decision to speak) should be separate from "state tracking" (remembering what the user wants). The Harness handles the "what," allowing the model to focus on the "when" and "how."
- The "Implicit" Difficulty Gap: Evaluation showed that implicit and dynamic intents are significantly harder for current AI than explicit ones. Agents often struggle to determine if a user is actually "stuck" or just moving slowly.
Key Results
- Performance Boost: StreamVibe-VL improved response timing and answer quality by 10.7% over other open-source baselines.
- Harness Gains: Adding the StreamVibe-Harness state-tracking system resulted in an additional 12.3% gain in proactive interaction stability, particularly by reducing redundant advice.
- Generalization: The model maintained strong performance on general video understanding benchmarks (like Video-MME), proving that learning to be proactive doesn't degrade basic perception skills.
Limitations and Future Work
The authors acknowledge that real-world interaction is even more complex than their benchmark. Future work should address:
- Co-evolving Interactions: In the real world, an agent's advice changes the user's behavior, which in turn might change the user's next intent. This feedback loop is not yet fully captured.
- Richer Intent Evolution: While the paper covers revisions and interruptions, more complex "negotiations" between user and agent over long horizons remain a challenge.
- Memory Efficiency: While hierarchical memory helps, managing extremely long-horizon tasks (hours of video) without losing critical fine-grained details is still an open research question.
Summary updated