TL;DR
ALIVE transforms streaming video LLMs from passive observers into proactive, interactive assistants capable of managing evolving tasks and seeking user feedback. By employing a two-stage training process—contrastive supervised fine-tuning and reinforcement learning with a novel "Behavior-Aligned Credit Assignment" (BACA)—the model learns the critical balance of when to speak, when to stay silent, and how to ask for missing information.
What Problem Does This Paper Address?
While recent "streaming" video models can process continuous visual data, they remain largely passive. They typically respond to user queries in isolation and lack the "situational awareness" to maintain long-term tasks or engage the user as a scene unfolds.
For instance, if you ask a standard model to "tell me when the person pets the rat," it might wait for that event. But if you change your mind mid-video ("Actually, tell me when they sneeze instead"), most models struggle to coordinate these evolving instructions. Furthermore, if the camera view is blocked, a passive model simply fails to answer, whereas an interactive assistant should proactively ask the user to "move the camera" or "lift the lid." ALIVE bridges this gap, moving us closer to the vision of a JARVIS-like multimodal AI.
Methods
The authors introduce a two-stage training framework to endow the model with interactive capabilities:
- Contrastive Supervised Fine-Tuning (SFT): The model is trained on pairs of "contrasting" scenarios. Response-gate contrasts teach the model to stay silent (S) or respond (R) based on task status (e.g., the same visual event should trigger a response if the task is active, but silence if it was recently cancelled). Action-strategy contrasts teach the model to choose between answering directly or asking for clarification based on how much evidence is currently visible.
- Reinforcement Learning with BACA: Standard RL often struggles with "credit assignment" in long video streams—it's hard to know which specific action led to a successful interaction. The authors propose Behavior-Aligned Credit Assignment (BACA). This method breaks a long interaction into "behavioral units" (like strategy selection or task response) and calculates rewards for both the overall trajectory and these local units. This ensures the model is penalized for redundant talking while being rewarded for "critical silence."
Datasets
The researchers curated two massive new datasets for this task:
- ALIVE-SFT-288K: A large-scale dataset for supervised learning covering proactive monitoring and task coordination.
- ALIVE-RL-3K: A high-quality set of interaction trajectories for reinforcement learning.
They also introduced FSI-Bench (Feedback-Seeking Interaction Benchmark), a new evaluation tool specifically designed to measure how well a model can identify missing prerequisites and ask the user for help (e.g., "Ask-for-Monitor," "Ask-for-Information," or "Ask-for-Execution").
Key Insights and Ideas
- Silence is a Feature, Not a Bug: In streaming video, knowing when not to talk is as important as knowing what to say. ALIVE explicitly supervises "critical silence" to prevent the model from becoming a "chatterbox" that provides redundant notifications.
- Context Over Pixels: The paper demonstrates that the "correct" behavior is often determined by the interaction history rather than the current video frame. By using contrastive pairs, the model learns that its response depends on whether a task was previously modified or fulfilled.
- Proactive Feedback Seeking: Instead of hallucinating or failing when evidence is missing, ALIVE learns to "delegate" tasks back to the user (e.g., "I can't read the map yet, please hold it closer"). This creates a bidirectional communication loop.
Key Results
ALIVE achieved state-of-the-art (SOTA) performance across four major public benchmarks and the new FSI-Bench:
- IPIBench: ALIVE scored 40.3, a massive leap over the previous best model, AURA (21.6), showing its superior ability to handle interleaved and updated requests.
- StreamingBench: It reached 76.0, outperforming specialized models like VideoChat3.
- FSI-Bench: ALIVE significantly outperformed proprietary models like GPT-4o (referred to as GPT-6 Astra/Sol in the paper's evaluation) in streaming interaction accuracy, particularly in "Ask-for-Information" and "Ask-for-Execution" tasks.
Limitations and Future Work
Despite its strong performance, ALIVE is currently a vision-language model that communicates via text. Real-world interaction often requires speech, so the authors note that integrating Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) is a necessary next step. Furthermore, while ALIVE is the top performer, there is still a significant gap between its scores and human-level performance on the FSI-Bench, suggesting that "common sense" reasoning in dynamic environments remains a challenging frontier. Future work will likely focus on "native omni-modal" models that process vision, audio, and language simultaneously.
Summary updated