Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, Jie ZhouarXiv 2020 · OpenReview

Abstract

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

Introduction

Imagine chatting with an AI about a video you’re both watching. To be a good conversationalist, the AI needs to understand the visual action, the background noise, the previous dialogue turns, and the overall summary of the scene. This is the Audio-Visual Scene-Aware Dialog (AVSD) task.

The core challenge is "multimodal fusion": how do you get a model to effectively combine high-dimensional video/audio data with discrete text tokens? Previous approaches often used separate encoders for each modality and then tried to "glue" them together using attention mechanisms. However, these methods often fail to capture the deep, joint representations necessary for truly fluent and context-aware responses.

Methodology: The Universal Multimodal Transformer

The proposed Universal Multimodal Transformer is based on the GPT-2 architecture. Instead of separate encoders, all modalities are projected into a shared embedding space and concatenated into one long sequence.

The Universal Multimodal Transformer architecture treats video, audio, and text as a single sequence.

Key Components:

  • Unified Architecture: The model uses a 12-layer decoder-only Transformer. It takes a sequence consisting of: [Video-Audio Features] + [Caption] + [Dialogue History] + [Current Question] + [Response].
  • Feature Extraction: Video features are extracted using I3D (RGB and Optical Flow) to capture motion and objects, while audio features are extracted using VGGish.
  • Multi-task Learning: To force the model to "understand" the relationship between these inputs, it is trained on three simultaneous tasks:
  • Response Language Modeling (RLM): The primary goal—predicting the next word in the dialogue response.
  • Video-Audio Sequence Modeling (VASM): A self-supervised regression task where the model predicts the next video-audio feature in a sequence using L2 loss. This helps the model learn temporal dynamics.
  • Caption Language Modeling (CLM): Generating the video's caption based on the video-audio features.

Experimental Results

The model dominated the DSTC8-AVSD challenge, achieving state-of-the-art results across various metrics. On the DSTC7 test set, it significantly outperformed previous state-of-the-art models like MTN and Hierarchical Attention.

Comparison of the proposed model against baseline and state-of-the-art methods on the DSTC7-AVSD dataset.
  • Objective Metrics: The model achieved a BLEU-4 score of 0.459 and a CIDEr score of 1.308 on the DSTC7 test set.
  • Human Evaluation: In the DSTC8 challenge, the model received a human rating of 3.934 (out of 5), the highest among all participants and very close to the ground-truth human references (scored at 4.0).
  • Multimodal Synergy: Adding the video-audio sequence modeling (VASM) task consistently improved dialogue quality, proving that the model's "understanding" of the video directly translates to better conversation.

Key Insights

  • Pre-training is King: This was the first work to apply pre-trained natural language generation models (like GPT-2) to multimodal dialogue. The linguistic "common sense" inside GPT-2 provided a massive head start.
  • Single-Stream Fusion: By treating video features as "pseudo-tokens" and feeding them directly into the Transformer alongside text, the model can use its self-attention layers to learn complex cross-modal relationships automatically.
  • Optimal Context: Through ablation studies, the authors found that a dialogue history of 3 turns was the "sweet spot." Too little context leads to confusion; too much introduces noise.

Conclusion and Future Work

While the model excels at using video to answer questions, it still faces challenges in "recaptioning"—generating high-quality captions from scratch using only video features. This suggests a limited ability to extract deep semantic summaries from raw pixels compared to its conversational prowess. Future work will explore end-to-end training to learn directly from raw data and better video comprehension techniques for "zero-knowledge" scenarios where captions are unavailable.

Summary updated

Abstract

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.
BibTeX
@misc{li2020bridging,
  author = {Zekang Li and Zongjia Li and Jinchao Zhang and Yang Feng and Cheng Niu and Jie Zhou},
  title = {{Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog}},
  year = {2020},
  eprint = {2002.00163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.00163},
}
Figure