StreamVibe: Proactive Video Agents for Evolving User Intent

Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou, Yue Cao, Haonan Wang, Ruixi Fan, Lan Wang, Sheng Liu, Songyou Peng, Yale Song, Kevin Qinghong Lin, Xin Eric Wang, Jui-Hsin Lai, Youbao Tang

TL;DR

StreamVibe introduces a framework and benchmark for proactive video agents that don't just wait for commands, but infer implicit and evolving user intents. By combining a streaming vision-language model (StreamVibe-VL) with a state-tracking "Harness," the system achieves a 10.7% improvement in response timing and quality over existing baselines.

What Problem Does This Paper Address?

Most current AI assistants are "passive"—they wait for an explicit request like "Set a timer" or "Notify me when the water boils." However, in complex real-world tasks (like cooking a new recipe or assembling furniture), a truly helpful agent needs to understand implicit intent (inferring help is needed because the user looks stuck) and dynamic intent (adapting when the user changes their mind or adds new constraints mid-task).

Existing models struggle with three main gaps:

  1. Reliance on explicit requests: They miss opportunities to help when the user hasn't asked a direct question.
  2. Insufficient state tracking: They often lose track of what has already been accomplished, leading to redundant or outdated advice.
  3. Limited adaptation: They fail to update their internal "to-do list" when user goals shift during an interaction.
Gaps in existing interaction paradigms

Methods

The authors propose a two-part solution: a specialized model and an agentic framework.

1. StreamVibe-VL (The Model)

This is a streaming proactive model trained in two stages:

  • Adaptive Streaming SFT: A supervised fine-tuning stage that teaches the model to balance "waiting" (observing) and "speaking" (responding). It uses weighted tokens to ensure the model doesn't become over-talkative or too silent.
  • Localized Trajectory RL: Using Group Relative Policy Optimization (GRPO), the model is refined specifically around "help points." This avoids the "noise" of a full video and focuses the learning on critical moments where timing and content quality matter most.

2. StreamVibe-Harness (The Framework)

To handle long-term interactions, the authors developed a "Harness" that wraps around the model:

  • Intent Lifecycle Management: It treats every user intent as a persistent state that can be active, satisfied, or revised.
  • Hierarchical Memory: Instead of cramming the entire video history into the model's context window, the Harness uses a memory system that keeps recent observations fine-grained while progressively compressing older information.
Overview of the StreamVibe framework

Datasets

The researchers introduced StreamVibe-Bench, a comprehensive testbed for proactive interaction:

  • Scale: 4.1K evaluation samples and 85K training samples.
  • Diversity: Covers both physical (egocentric daily activities) and digital (GUI/computer use) environments.
  • Intent Levels: It categorizes tasks into Explicit (monitoring a specific event), Implicit (inferring help from high-level goals), and Dynamic (handling revisions, compositions, or interruptions).
StreamVibe-Bench statistics and sample allocation

Key Insights and Ideas

  • Proactivity is a Timing Problem: It’s not just about what to say, but when to say it. The paper shows that optimizing for "help points" via RL is much more effective than standard video-captioning approaches.
  • State Tracking vs. Policy: The authors found that a model's "policy" (the decision to speak) should be separate from "state tracking" (remembering what the user wants). The Harness handles the "what," allowing the model to focus on the "when" and "how."
  • The "Implicit" Difficulty Gap: Evaluation showed that implicit and dynamic intents are significantly harder for current AI than explicit ones. Agents often struggle to determine if a user is actually "stuck" or just moving slowly.

Key Results

  • Performance Boost: StreamVibe-VL improved response timing and answer quality by 10.7% over other open-source baselines.
  • Harness Gains: Adding the StreamVibe-Harness state-tracking system resulted in an additional 12.3% gain in proactive interaction stability, particularly by reducing redundant advice.
  • Generalization: The model maintained strong performance on general video understanding benchmarks (like Video-MME), proving that learning to be proactive doesn't degrade basic perception skills.
Performance gains from policy learning and the Harness

Limitations and Future Work

The authors acknowledge that real-world interaction is even more complex than their benchmark. Future work should address:

  • Co-evolving Interactions: In the real world, an agent's advice changes the user's behavior, which in turn might change the user's next intent. This feedback loop is not yet fully captured.
  • Richer Intent Evolution: While the paper covers revisions and interruptions, more complex "negotiations" between user and agent over long horizons remain a challenge.
  • Memory Efficiency: While hierarchical memory helps, managing extremely long-horizon tasks (hours of video) without losing critical fine-grained details is still an open research question.

Summary updated

Abstract

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.
BibTeX
@misc{liu2026streamvibe,
  author = {Chengzhi Liu and Yuxiang Lai and Yuzhe Yang and Yanqing Liu and Jiheng Hou and Yue Cao and Haonan Wang and Ruixi Fan and Lan Wang and Sheng Liu and Songyou Peng and Yale Song and Kevin Qinghong Lin and Xin Eric Wang and Jui-Hsin Lai and Youbao Tang},
  title = {{StreamVibe: Proactive Video Agents for Evolving User Intent}},
  year = {2026},
  eprint = {2610.34662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34662},
}
Figure