TL;DR
AURA 2.0 is a streaming Video Large Language Model (VideoLLM) that shifts AI assistants from reactive to proactive. By introducing Model-Initiated Conversation (MIC), Stream-Aware Tool Calling (SATC), and Interactive Visual Grounding (IVG), the model can now start conversations about interesting visual events, use external tools to verify facts, and point to specific objects in a video stream—all in real-time.
What Problem Does This Paper Address?
Most current AI assistants are "query-driven": they sit silently until a user asks a question. In the context of streaming video (like a smart home camera or a wearable assistant), this is a missed opportunity. A truly helpful assistant should be able to notice a pot boiling over, a rare bird flying by, or a specific person entering a room and initiate a conversation without being prompted.
The technical challenges are significant:
- Timing: When is a visual event "worth" talking about?
- Redundancy: How does the model avoid repeating itself or stating the obvious?
- Memory: How can a model remember something that happened 10 minutes ago in a continuous stream without running out of memory?
- Grounding: How can the model precisely link its words to moving objects in the video?
Methods
AURA 2.0 builds on the original AURA architecture, which processes video in temporal chunks. The authors introduced several key architectural and training innovations:
- Model-Initiated Conversation (MIC): The model is trained to generate a
<|respond|>token even when the user input field is empty, triggered by salient visual events. - Long-Horizon Interaction Memory (LHIM): To handle long videos, AURA 2.0 uses a hierarchical memory. As video frames leave the "active window" (the last 30 seconds), an auxiliary model summarizes them into compact textual descriptions and QA pairs. This allows the model to "remember" past events without storing every frame.
- Stream-Aware Tool Calling (SATC): The model can pause its response to invoke external APIs (like web search or a calculator) to incorporate real-time information into its streaming dialogue.
- Interactive Visual Grounding (IVG): The model can accept user-marked regions as input and produce frame-aligned bounding boxes in its output text.
- Three-Stage Training Curriculum:
- Foundation: General streaming QA and tool use.
- Interaction: Introducing MIC and IVG.
- Refinement: Capability composition (e.g., using a tool and initiating a conversation).
Datasets
The researchers constructed a massive dedicated streaming interaction dataset:
- Scale: 11.4 billion tokens, 5,400 hours of video, and 40 million training frames.
- Composition: It combines inherited QA tasks with new data for MIC (MLLM-generated conversation openings), SATC (real API execution records), and IVG (sourced from grounding datasets like MeViS and Refer-YouTube-VOS).
- Benchmarks: Evaluation was performed on four major public streaming benchmarks: IPIBench, StreamArena, ProactiveVideoQA, and PhoStream.
Key Insights and Ideas
- The "Silence" Imbalance: In a streaming setting, the model should be silent ~95% of the time. Standard cross-entropy loss fails here because the model learns to always be silent. The authors used focal-loss-based supervision on response-control tokens to force the model to care about the rare moments it actually needs to speak.
- Textual Memory is Scalable: Instead of trying to compress visual features into a "memory bank," summarizing events into text (LHIM) proved highly effective for long-term reasoning while keeping the inference cost low.
- Context-Aware Proactivity: AURA 2.0 doesn't just react to pixels; it reacts to the history. If a user already mentioned they are looking for their keys, the model uses that context to decide that "seeing keys on the table" is a high-priority event for MIC.
Key Results
AURA 2.0 achieves state-of-the-art results among open-source streaming models:
- IPIBench: Achieved a timing accuracy of 49.30% in proactive monitoring, outperforming Qwen3.5-9B and approaching proprietary models like GPT-4o (50.59%).
- PhoStream: Scored 65.69 overall, exceeding the strongest proprietary baseline (Doubao-Seed-1.8) by nearly 10 points.
- SATC: Enabling tools improved question-answering accuracy from 35.9% to 48.5%.
- Efficiency: The model is highly optimized for real-time use. With a single session, the Time to First Token (TTFT) is just 157ms. Even with 16 concurrent sessions, it maintains a Real-Time Factor (RTF) well below 1.0, meaning it processes video faster than it arrives.
Limitations and Future Work
The authors acknowledge that "what is worth discussing" is inherently subjective and depends on individual user preferences. Currently, the system uses a cascaded pipeline (ASR → VideoLLM → TTS).
Future work aims to move toward a native omni-modal, full-duplex model. This would allow the assistant to listen and speak simultaneously—for example, providing real-time guidance during piano practice where it can "hear" a wrong note and "see" a fingering error at the same instant.
Summary updated