TL;DR
This paper introduces a novel "video-text duet" interaction format for Video Large Language Models (VideoLLMs), allowing for real-time, turn-based conversations about videos as they play. They propose a new dataset (MMDuetIT) and a model (MMDuet) that excels at time-sensitive video comprehension tasks, demonstrating significant improvements in areas like dense video captioning and temporal video grounding.
What Problem Does This Paper Address?
Current VideoLLMs typically operate on a "whole video" interaction model: the user provides the entire video and a query, and the model responds only after processing the whole input. This approach is ill-suited for real-time applications like live-streaming comprehension, where timely responses are crucial, and also hinders performance on time-sensitive tasks that require precise localization of video segments. The authors aim to bridge this gap by enabling VideoLLMs to engage in a more dynamic, interactive dialogue with users, mirroring a duet where both parties contribute concurrently.
Methods
The core innovation is the Video-Text Duet Interaction Format. This format allows both the user and the model to interject text messages at any point during continuous video playback. The video is processed frame-by-frame, and the model can decide to respond after each frame. To achieve this, the authors introduce two new components to their model, MMDuet:
- Informative Head: Predicts how much new information is acquired from a given frame.
- Relevance Head: Predicts whether the current frame is relevant to the user's query.
These heads help MMDuet determine when to speak. The model architecture itself is a standard transformer-decoder-based LLM augmented with visual encoders.
Datasets
- MMDuetIT: A newly constructed dataset designed to train VideoLLMs for the video-text duet interaction format. It comprises three tasks:
- Dense Captioning (using data from Shot2Story and COIN)
- Multi-Answer Grounded Video Question Answering (MAGQA) (newly proposed task, data generated using GPT40)
- Temporal Video Grounding (using data from DiDeMo, HiRESTgrounding, and QueryD)
- Shot2Story: Used for dense captioning training.
- COIN: Used for dense captioning training.
- DiDeMo, HiRESTgrounding, QueryD: Used for temporal video grounding tasks.
- StreamingBench: Used to evaluate proactive output on streaming videos.
Key Insights and Ideas
- The "Duet" is Key: The paper strongly argues that the interaction format is as crucial as model architecture for real-world VideoLLM applications. The duet format unlocks real-time capabilities.
- When to Speak Matters: The introduction of "informative" and "relevance" heads is a clever way to equip LLMs with the ability to self-regulate their responses in a temporal context, moving beyond simple token prediction.
- Time-Sensitive Tasks Benefit Greatly: The duet format and the model's ability to respond in real-time lead to significant performance gains on tasks that inherently require temporal understanding and localization.
- Dataset Design is Crucial: The MMDuetIT dataset is specifically designed to teach models the nuances of this new interaction, including how to generate informative labels for training the decision-making heads.
Key Results
MMDuet, trained on MMDuetIT, demonstrates significant improvements across various time-sensitive tasks:
- Highlight Detection: 76% CIDEr improvement on YouCook2 dense video captioning.
- Temporal Video Grounding: 90% mAP on QVHighlights highlight detection and 25% [email protected] on Charades-STA temporal video grounding.
- Real-time Responses: MMDuet is shown to provide real-time replies, a capability lacking in baseline "whole video" models.
- Long Video Performance: MMDuet shows superior performance on longer videos, outperforming baselines that struggle with temporal localization and repetition.
- Proactive Output: MMDuet outperforms existing streaming VideoLLMs on the Proactive Output task of StreamingBench.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Hyperparameter Sensitivity: While the
need_responsecriterion is shown to be robust, it still requires tuning. - Future Frame Information: The current model doesn't incorporate information from subsequent frames when generating real-time responses, which could be crucial for predicting future events.
- Inference Speed: While improved, further optimization for inference speed is needed, especially for avoiding duplicate responses.
- Real-world Datasets: More datasets capturing longer, real-world live-streaming scenarios are needed.
- Multi-turn Questioning: The current MAGQA task assumes a single question per video; extending this to handle arbitrary, multi-turn questions is a future direction.
Summary updated