Multi-Speaker Video Dialog with Frame-Level Temporal Localization

Qiang Wang, Pin Jiang, Zhiyi Guo, Yahong Han, Zhou Zhao

Abstract

Traditional scene-aware dialog systems often operate under the assumption that the relevant video context is pre-segmented and provided to the model. However, real-world interactions, such as those in live streaming chats, involve multiple speakers discussing specific moments within long, unedited video streams. This paper introduces the task of Multi-Speaker Video Dialog with Frame-Level Temporal Localization (MSVD-TL). We propose a multi-task learning framework that simultaneously predicts the next response and localizes the relevant video segment. Our approach utilizes a Selective Refinement module to align multimodal features, demonstrating that temporal localization is not just a sub-task but a critical component for improving dialog accuracy.

Introduction

In the evolution of visual dialog, moving from static images to video has introduced significant complexity. While "Scene-Aware Dialog" has made strides, it typically relies on manually cropped video clips. In practical scenarios—like a group of friends discussing a 90-minute soccer match—the AI must identify which specific event (e.g., a goal or a foul) is being discussed without explicit timestamps. Furthermore, real-world dialog is often multi-speaker and informal, diverging from the structured Q&A formats of existing datasets.

Overview of the MSVD-TL task, illustrating the need to identify relevant video sub-segments within a continuous stream.

The MSVD-TL Framework

To address these challenges, we present a multi-task framework designed to handle the chaotic nature of multi-speaker streams. The core of our model is the Selective Refinement Module, which employs cross-attention to filter out irrelevant "background" frames and focus on the semantically significant actions that trigger conversation.

Methodology

The architecture consists of three primary stages:

  1. Feature Encoding: Video frames are processed via Inception-v3, while chat history and candidate responses are encoded using GloVe embeddings and LSTMs to capture temporal dependencies.
  2. Selective Refinement: Instead of simple concatenation, the model uses cross-attention. The video features refine the chat history representation, and both together refine the representation of the candidate response. This ensures the model attends to the specific words and frames that are mutually relevant.
  3. Multi-Task Prediction: The refined representations feed into two parallel heads: a Response Prediction head for ranking candidates and a Temporal Localization head that assigns relevance scores to every frame in the video fragment.
The proposed multi-task architecture featuring Selective Refinement and parallel heads for localization and response prediction.

Experimental Evaluation

We evaluated our model on an extended version of the Twitch-FIFA dataset, comprising over 10,000 training samples from live FIFA soccer matches. The task involves processing 50-second video fragments to predict responses from a 10-second window while localizing the 20-second segment the chat refers to.

Key Results

Our findings highlight a synergistic relationship between localization and dialog:

  • Performance Gains: On the standard 20-second task, our "SelRef" model achieved a state-of-the-art R@1 of 21.4%.
  • The Localization Boost: When faced with longer 50-second fragments, enabling the Temporal Localization module improved the R@1 score from 20.9% to 21.5%. This proves that knowing when something happened directly informs what should be said.
  • Qualitative Alignment: Visualizations confirm that the localization head's predicted relevance scores closely align with ground-truth segments, effectively "watching" the right part of the game to understand the context.

Conclusion

The MSVD-TL task represents a significant step toward more realistic AI assistants capable of navigating long-form video content. By integrating temporal localization with multi-speaker dialog, our framework demonstrates that understanding the temporal context of a conversation is essential for generating relevant and scene-aware responses. Future work will explore generalizing this approach to more continuous video domains and moving from retrieval-based to generative response models.

Summary updated

Abstract

To simulate human interaction in real life, dialog systems are introduced to generate a response to previous chat utterances. There have been several studies for two-speaker video dialogs in the form of question answering. However, more informative semantic cues might be exploited via a multi-rounds chatting or discussing about the video among multiple speakers. So multi-speakers video dialogs are more applicable in real life. Besides, speakers always chat about a sub-segment of the long video fragment for a period of time. Current video dialog systems require to be directly given the relevant video sub-segment which speakers are chatting about. However, it is always hard to accurately spot the corresponding video sub-segment in practical applications. In this paper, we introduce a novel task of Multi-Speaker Video Dialog with frame-level Temporal Localization (MSVD-TL) to make video dialog systems more applicable. Given a long video fragment and a set of chat history utterances, MSVD-TL targets to predict the following response and localize the relevant video sub-segment in frame level, simultaneously. We develop a new multi-task model with a response prediction module and a frame-level temporal localization module. Besides, we focus on the characteristic of the video dialog generation process and exploit the relation among the video fragment, the chat history, and the following response to refine their representations. We evaluate our approach for both the Multi-Speaker Video Dialog without frame-level temporal localization (MSVD w/o TL) task and the MSVD-TL task. The experimental results further demonstrate that MSVD-TL enhances the applicability of video dialog in real life.
BibTeX
@inproceedings{wang2020multi,
  author = {Qiang Wang and Pin Jiang and Zhiyi Guo and Yahong Han and Zhou Zhao},
  title = {{Multi-Speaker Video Dialog with Frame-Level Temporal Localization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6901},
}
Figure