ALIVE: Making Streaming Video Models Interactive

Jinzhao Li, Yuan Xu, Jinxuan Wu, Yukun Wang, Yijia Lei, Hanyu Ji, Yinuo Chen, Zining Huang, Chi Zhang, Hang Ye, Yiwu Zhong, Honglei Yan, Panwang Pan, Yizhou Wang, Yin Li, Miao LiuarXiv 2026 · arXiv

TL;DR

ALIVE transforms streaming video LLMs from passive observers into proactive, interactive assistants capable of managing evolving tasks and seeking user feedback. By employing a two-stage training process—contrastive supervised fine-tuning and reinforcement learning with a novel "Behavior-Aligned Credit Assignment" (BACA)—the model learns the critical balance of when to speak, when to stay silent, and how to ask for missing information.

What Problem Does This Paper Address?

While recent "streaming" video models can process continuous visual data, they remain largely passive. They typically respond to user queries in isolation and lack the "situational awareness" to maintain long-term tasks or engage the user as a scene unfolds.

For instance, if you ask a standard model to "tell me when the person pets the rat," it might wait for that event. But if you change your mind mid-video ("Actually, tell me when they sneeze instead"), most models struggle to coordinate these evolving instructions. Furthermore, if the camera view is blocked, a passive model simply fails to answer, whereas an interactive assistant should proactively ask the user to "move the camera" or "lift the lid." ALIVE bridges this gap, moving us closer to the vision of a JARVIS-like multimodal AI.

Methods

The authors introduce a two-stage training framework to endow the model with interactive capabilities:

  1. Contrastive Supervised Fine-Tuning (SFT): The model is trained on pairs of "contrasting" scenarios. Response-gate contrasts teach the model to stay silent (S) or respond (R) based on task status (e.g., the same visual event should trigger a response if the task is active, but silence if it was recently cancelled). Action-strategy contrasts teach the model to choose between answering directly or asking for clarification based on how much evidence is currently visible.
  2. Reinforcement Learning with BACA: Standard RL often struggles with "credit assignment" in long video streams—it's hard to know which specific action led to a successful interaction. The authors propose Behavior-Aligned Credit Assignment (BACA). This method breaks a long interaction into "behavioral units" (like strategy selection or task response) and calculates rewards for both the overall trajectory and these local units. This ensures the model is penalized for redundant talking while being rewarded for "critical silence."
The two-stage training pipeline: Contrastive SFT followed by RL with BACA.

Datasets

The researchers curated two massive new datasets for this task:

  • ALIVE-SFT-288K: A large-scale dataset for supervised learning covering proactive monitoring and task coordination.
  • ALIVE-RL-3K: A high-quality set of interaction trajectories for reinforcement learning.

They also introduced FSI-Bench (Feedback-Seeking Interaction Benchmark), a new evaluation tool specifically designed to measure how well a model can identify missing prerequisites and ask the user for help (e.g., "Ask-for-Monitor," "Ask-for-Information," or "Ask-for-Execution").

Comparison of ALIVE datasets against existing streaming video training sets.

Key Insights and Ideas

  • Silence is a Feature, Not a Bug: In streaming video, knowing when not to talk is as important as knowing what to say. ALIVE explicitly supervises "critical silence" to prevent the model from becoming a "chatterbox" that provides redundant notifications.
  • Context Over Pixels: The paper demonstrates that the "correct" behavior is often determined by the interaction history rather than the current video frame. By using contrastive pairs, the model learns that its response depends on whether a task was previously modified or fulfilled.
  • Proactive Feedback Seeking: Instead of hallucinating or failing when evidence is missing, ALIVE learns to "delegate" tasks back to the user (e.g., "I can't read the map yet, please hold it closer"). This creates a bidirectional communication loop.
Examples of ALIVE performing proactive monitoring, task coordination, and feedback-seeking.

Key Results

ALIVE achieved state-of-the-art (SOTA) performance across four major public benchmarks and the new FSI-Bench:

  • IPIBench: ALIVE scored 40.3, a massive leap over the previous best model, AURA (21.6), showing its superior ability to handle interleaved and updated requests.
  • StreamingBench: It reached 76.0, outperforming specialized models like VideoChat3.
  • FSI-Bench: ALIVE significantly outperformed proprietary models like GPT-4o (referred to as GPT-6 Astra/Sol in the paper's evaluation) in streaming interaction accuracy, particularly in "Ask-for-Information" and "Ask-for-Execution" tasks.

Limitations and Future Work

Despite its strong performance, ALIVE is currently a vision-language model that communicates via text. Real-world interaction often requires speech, so the authors note that integrating Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) is a necessary next step. Furthermore, while ALIVE is the top performer, there is still a significant gap between its scores and human-level performance on the FSI-Bench, suggesting that "common sense" reasoning in dynamic environments remains a challenging frontier. Future work will likely focus on "native omni-modal" models that process vision, audio, and language simultaneously.

Summary updated

Abstract

Streaming video LLMs can continuously observe the world, yet often handle user requests in isolation. We introduce ALIVE, an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction. We construct ALIVE-SFT-288K and ALIVE-RL-3K for two-stage training. Contrastive supervised fine-tuning teaches when and how to respond, while reinforcement learning with Behavior-Aligned Credit Assignment (BACA) combines trajectory-level and local advantages to refine decisions, responses, and silence across multi-turn interactions. We also introduce FSI-Bench to evaluate feedback seeking and subsequent task completion. ALIVE achieves the highest overall scores among the compared models on four public benchmarks and FSI-Bench, demonstrating strong streaming understanding and interactive capabilities.
BibTeX
@misc{li2026alive,
  author = {Jinzhao Li and Yuan Xu and Jinxuan Wu and Yukun Wang and Yijia Lei and Hanyu Ji and Yinuo Chen and Zining Huang and Chi Zhang and Hang Ye and Yiwu Zhong and Honglei Yan and Panwang Pan and Yizhou Wang and Yin Li and Miao Liu},
  title = {{ALIVE: Making Streaming Video Models Interactive}},
  year = {2026},
  eprint = {2611.21366},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21366},
}
Figure