AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video

Shuhan Li, Huankang Guan, Xintong Guo, Yang Bo, Yuyang He, Dunyuan Xu, Jingyan Chen, Fang Liu, Tsz Kin Wan, Jinpeng Chen, Zhenqi He, Yuxuan Hu, Wenbo Li, Han Gao, Qijun Li, Long Chen, Hongsheng Li, Rui LiuarXiv 2026 · arXiv

TL;DR

AURA 2.0 is a streaming Video Large Language Model (VideoLLM) that shifts AI assistants from reactive to proactive. By introducing Model-Initiated Conversation (MIC), Stream-Aware Tool Calling (SATC), and Interactive Visual Grounding (IVG), the model can now start conversations about interesting visual events, use external tools to verify facts, and point to specific objects in a video stream—all in real-time.

What Problem Does This Paper Address?

Most current AI assistants are "query-driven": they sit silently until a user asks a question. In the context of streaming video (like a smart home camera or a wearable assistant), this is a missed opportunity. A truly helpful assistant should be able to notice a pot boiling over, a rare bird flying by, or a specific person entering a room and initiate a conversation without being prompted.

The technical challenges are significant:

  1. Timing: When is a visual event "worth" talking about?
  2. Redundancy: How does the model avoid repeating itself or stating the obvious?
  3. Memory: How can a model remember something that happened 10 minutes ago in a continuous stream without running out of memory?
  4. Grounding: How can the model precisely link its words to moving objects in the video?

Methods

AURA 2.0 builds on the original AURA architecture, which processes video in temporal chunks. The authors introduced several key architectural and training innovations:

  • Model-Initiated Conversation (MIC): The model is trained to generate a <|respond|> token even when the user input field is empty, triggered by salient visual events.
  • Long-Horizon Interaction Memory (LHIM): To handle long videos, AURA 2.0 uses a hierarchical memory. As video frames leave the "active window" (the last 30 seconds), an auxiliary model summarizes them into compact textual descriptions and QA pairs. This allows the model to "remember" past events without storing every frame.
  • Stream-Aware Tool Calling (SATC): The model can pause its response to invoke external APIs (like web search or a calculator) to incorporate real-time information into its streaming dialogue.
  • Interactive Visual Grounding (IVG): The model can accept user-marked regions as input and produce frame-aligned bounding boxes in its output text.
  • Three-Stage Training Curriculum:
    1. Foundation: General streaming QA and tool use.
  • Interaction: Introducing MIC and IVG.
  • Refinement: Capability composition (e.g., using a tool and initiating a conversation).
Figure 2. Overview of the AURA 2.0 streaming interaction framework. The model incrementally processes video chunks together with user inputs and interaction history, producing either <|silent|> for silent observation or <|respond|> followed by a text response or tool call. Tool results are incorporated into the context to support subsequent responses. Dashed outlines indicate historical messages beyond the active visual window.

Datasets

The researchers constructed a massive dedicated streaming interaction dataset:

  • Scale: 11.4 billion tokens, 5,400 hours of video, and 40 million training frames.
  • Composition: It combines inherited QA tasks with new data for MIC (MLLM-generated conversation openings), SATC (real API execution records), and IVG (sourced from grounding datasets like MeViS and Refer-YouTube-VOS).
  • Benchmarks: Evaluation was performed on four major public streaming benchmarks: IPIBench, StreamArena, ProactiveVideoQA, and PhoStream.
Figure 4. Training data composition and response-control statistics across the three training stages. (a) Sample distribution by stage (inner ring) and task (outer ring). (b) Stage-wise sample and token allocations and silent/response decision proportions. (c) Task-wise sample counts and shares aggregated across stages. Together, these distributions characterize the task coverage and supervision balance of the training mixture.

Key Insights and Ideas

  • The "Silence" Imbalance: In a streaming setting, the model should be silent ~95% of the time. Standard cross-entropy loss fails here because the model learns to always be silent. The authors used focal-loss-based supervision on response-control tokens to force the model to care about the rare moments it actually needs to speak.
  • Textual Memory is Scalable: Instead of trying to compress visual features into a "memory bank," summarizing events into text (LHIM) proved highly effective for long-term reasoning while keeping the inference cost low.
  • Context-Aware Proactivity: AURA 2.0 doesn't just react to pixels; it reacts to the history. If a user already mentioned they are looking for their keys, the model uses that context to decide that "seeing keys on the table" is a high-priority event for MIC.

Key Results

AURA 2.0 achieves state-of-the-art results among open-source streaming models:

  • IPIBench: Achieved a timing accuracy of 49.30% in proactive monitoring, outperforming Qwen3.5-9B and approaching proprietary models like GPT-4o (50.59%).
  • PhoStream: Scored 65.69 overall, exceeding the strongest proprietary baseline (Doubao-Seed-1.8) by nearly 10 points.
  • SATC: Enabling tools improved question-answering accuracy from 35.9% to 48.5%.
  • Efficiency: The model is highly optimized for real-time use. With a single session, the Time to First Token (TTFT) is just 157ms. Even with 16 concurrent sessions, it maintains a Real-Time Factor (RTF) well below 1.0, meaning it processes video faster than it arrives.
Figure 1. Comparison of AURA 2.0 with MOSS-VL-Realtime and JoyAI-VL-Interaction across four public streaming benchmarks. The Tool use score for MOSS-VL-Realtime on StreamArena is plotted as zero because the model does not support tool calling.

Limitations and Future Work

The authors acknowledge that "what is worth discussing" is inherently subjective and depends on individual user preferences. Currently, the system uses a cascaded pipeline (ASR → VideoLLM → TTS).

Future work aims to move toward a native omni-modal, full-duplex model. This would allow the assistant to listen and speak simultaneously—for example, providing real-time guidance during piano practice where it can "hear" a wrong note and "see" a fingering error at the same instant.

Summary updated

Abstract

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.
BibTeX
@misc{li2026aura,
  author = {Shuhan Li and Huankang Guan and Xintong Guo and Yang Bo and Yuyang He and Dunyuan Xu and Jingyan Chen and Fang Liu and Tsz Kin Wan and Jinpeng Chen and Zhenqi He and Yuxuan Hu and Wenbo Li and Han Gao and Qijun Li and Long Chen and Hongsheng Li and Rui Liu},
  title = {{AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video}},
  year = {2026},
  eprint = {2611.17048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17048},
}
Figure