VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format

Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, June Wei, Hui Zhang, Dongyan Zhao2025 · DOI

TL;DR

This paper introduces a novel "video-text duet" interaction format for Video Large Language Models (VideoLLMs), allowing for real-time, turn-based conversations about videos as they play. They propose a new dataset (MMDuetIT) and a model (MMDuet) that excels at time-sensitive video comprehension tasks, demonstrating significant improvements in areas like dense video captioning and temporal video grounding.

What Problem Does This Paper Address?

Current VideoLLMs typically operate on a "whole video" interaction model: the user provides the entire video and a query, and the model responds only after processing the whole input. This approach is ill-suited for real-time applications like live-streaming comprehension, where timely responses are crucial, and also hinders performance on time-sensitive tasks that require precise localization of video segments. The authors aim to bridge this gap by enabling VideoLLMs to engage in a more dynamic, interactive dialogue with users, mirroring a duet where both parties contribute concurrently.

Methods

The core innovation is the Video-Text Duet Interaction Format. This format allows both the user and the model to interject text messages at any point during continuous video playback. The video is processed frame-by-frame, and the model can decide to respond after each frame. To achieve this, the authors introduce two new components to their model, MMDuet:

  • Informative Head: Predicts how much new information is acquired from a given frame.
  • Relevance Head: Predicts whether the current frame is relevant to the user's query.

These heads help MMDuet determine when to speak. The model architecture itself is a standard transformer-decoder-based LLM augmented with visual encoders.

Datasets

  • MMDuetIT: A newly constructed dataset designed to train VideoLLMs for the video-text duet interaction format. It comprises three tasks:
  • Dense Captioning (using data from Shot2Story and COIN)
    • Multi-Answer Grounded Video Question Answering (MAGQA) (newly proposed task, data generated using GPT40)
    • Temporal Video Grounding (using data from DiDeMo, HiRESTgrounding, and QueryD)
  • Shot2Story: Used for dense captioning training.
  • COIN: Used for dense captioning training.
  • DiDeMo, HiRESTgrounding, QueryD: Used for temporal video grounding tasks.
  • StreamingBench: Used to evaluate proactive output on streaming videos.

Key Insights and Ideas

  • The "Duet" is Key: The paper strongly argues that the interaction format is as crucial as model architecture for real-world VideoLLM applications. The duet format unlocks real-time capabilities.
  • When to Speak Matters: The introduction of "informative" and "relevance" heads is a clever way to equip LLMs with the ability to self-regulate their responses in a temporal context, moving beyond simple token prediction.
  • Time-Sensitive Tasks Benefit Greatly: The duet format and the model's ability to respond in real-time lead to significant performance gains on tasks that inherently require temporal understanding and localization.
  • Dataset Design is Crucial: The MMDuetIT dataset is specifically designed to teach models the nuances of this new interaction, including how to generate informative labels for training the decision-making heads.

Key Results

MMDuet, trained on MMDuetIT, demonstrates significant improvements across various time-sensitive tasks:

  • Highlight Detection: 76% CIDEr improvement on YouCook2 dense video captioning.
  • Temporal Video Grounding: 90% mAP on QVHighlights highlight detection and 25% [email protected] on Charades-STA temporal video grounding.
  • Real-time Responses: MMDuet is shown to provide real-time replies, a capability lacking in baseline "whole video" models.
  • Long Video Performance: MMDuet shows superior performance on longer videos, outperforming baselines that struggle with temporal localization and repetition.
  • Proactive Output: MMDuet outperforms existing streaming VideoLLMs on the Proactive Output task of StreamingBench.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Hyperparameter Sensitivity: While the need_response criterion is shown to be robust, it still requires tuning.
  • Future Frame Information: The current model doesn't incorporate information from subsequent frames when generating real-time responses, which could be crucial for predicting future events.
  • Inference Speed: While improved, further optimization for inference speed is needed, especially for avoiding duplicate responses.
  • Real-world Datasets: More datasets capturing longer, real-world live-streaming scenarios are needed.
  • Multi-turn Questioning: The current MAGQA task assumes a single question per video; extending this to handle arbitrary, multi-turn questions is a future direction.

Summary updated

BibTeX
@misc{wang2025videollm,
  author = {Yueqian Wang and Xiaojun Meng and Yuxuan Wang and Jianxin Liang and June Wei and Hui Zhang and Dongyan Zhao},
  title = {{VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.336},
}
Figure