StreamVibe: Proactive Video Agents for Evolving User Intent

Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou, Yue Cao, Haonan Wang, Ruixi Fan, Lan Wang, Sheng Liu, Songyou Peng, Yale Song, Kevin Qinghong Lin, Xin Eric Wang, Jui-Hsin Lai, Youbao Tang

Abstract

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while overlooking implicit user intents (e.g., "where should I click next?" when a user is navigating a website) and fail to continuously update their understanding as user intents evolve. These limitations make them less suitable for intent-aware interaction in real-world scenarios. To address these challenges, we introduce StreamVibe-Bench, a new benchmark that serves as a testbed for user intent understanding and interaction, comprising 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically adapts its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with flexible lifecycles through hierarchical memory, enabling it to track the latest user intents and environmental states. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding capabilities while improving response timing and answer quality by 10.7% over existing baselines; (ii) existing proactive models remain limited in understanding user intents and determining when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation during sustained interactions, resulting in a 12.3% gain in proactive interaction performance.
BibTeX
@misc{liu2026streamvibe,
  author = {Chengzhi Liu and Yuxiang Lai and Yuzhe Yang and Yanqing Liu and Jiheng Hou and Yue Cao and Haonan Wang and Ruixi Fan and Lan Wang and Sheng Liu and Songyou Peng and Yale Song and Kevin Qinghong Lin and Xin Eric Wang and Jui-Hsin Lai and Youbao Tang},
  title = {{StreamVibe: Proactive Video Agents for Evolving User Intent}},
  year = {2026},
  eprint = {2610.34662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34662},
}
Figure