Abstract
Traditional scene-aware dialog systems often operate under the assumption that the relevant video context is pre-segmented and provided to the model. However, real-world interactions, such as those in live streaming chats, involve multiple speakers discussing specific moments within long, unedited video streams. This paper introduces the task of Multi-Speaker Video Dialog with Frame-Level Temporal Localization (MSVD-TL). We propose a multi-task learning framework that simultaneously predicts the next response and localizes the relevant video segment. Our approach utilizes a Selective Refinement module to align multimodal features, demonstrating that temporal localization is not just a sub-task but a critical component for improving dialog accuracy.
Introduction
In the evolution of visual dialog, moving from static images to video has introduced significant complexity. While "Scene-Aware Dialog" has made strides, it typically relies on manually cropped video clips. In practical scenarios—like a group of friends discussing a 90-minute soccer match—the AI must identify which specific event (e.g., a goal or a foul) is being discussed without explicit timestamps. Furthermore, real-world dialog is often multi-speaker and informal, diverging from the structured Q&A formats of existing datasets.
The MSVD-TL Framework
To address these challenges, we present a multi-task framework designed to handle the chaotic nature of multi-speaker streams. The core of our model is the Selective Refinement Module, which employs cross-attention to filter out irrelevant "background" frames and focus on the semantically significant actions that trigger conversation.
Methodology
The architecture consists of three primary stages:
- Feature Encoding: Video frames are processed via Inception-v3, while chat history and candidate responses are encoded using GloVe embeddings and LSTMs to capture temporal dependencies.
- Selective Refinement: Instead of simple concatenation, the model uses cross-attention. The video features refine the chat history representation, and both together refine the representation of the candidate response. This ensures the model attends to the specific words and frames that are mutually relevant.
- Multi-Task Prediction: The refined representations feed into two parallel heads: a Response Prediction head for ranking candidates and a Temporal Localization head that assigns relevance scores to every frame in the video fragment.
Experimental Evaluation
We evaluated our model on an extended version of the Twitch-FIFA dataset, comprising over 10,000 training samples from live FIFA soccer matches. The task involves processing 50-second video fragments to predict responses from a 10-second window while localizing the 20-second segment the chat refers to.
Key Results
Our findings highlight a synergistic relationship between localization and dialog:
- Performance Gains: On the standard 20-second task, our "SelRef" model achieved a state-of-the-art R@1 of 21.4%.
- The Localization Boost: When faced with longer 50-second fragments, enabling the Temporal Localization module improved the R@1 score from 20.9% to 21.5%. This proves that knowing when something happened directly informs what should be said.
- Qualitative Alignment: Visualizations confirm that the localization head's predicted relevance scores closely align with ground-truth segments, effectively "watching" the right part of the game to understand the context.
Conclusion
The MSVD-TL task represents a significant step toward more realistic AI assistants capable of navigating long-form video content. By integrating temporal localization with multi-speaker dialog, our framework demonstrates that understanding the temporal context of a conversation is essential for generating relevant and scene-aware responses. Future work will explore generalizing this approach to more continuous video domains and moving from retrieval-based to generative response models.
Summary updated