Audio-Visual Scene-Aware Dialog and Reasoning Using Audio-Visual Transformers with Joint Student-Teacher Learning

Ankit Shah, Shijie Geng, Peng Gao, Anoop Cherian, Takaaki Hori, Tim K. Marks, Jonathan Le Roux, Chiori Hori

TL;DR

This paper introduces a more realistic framework for Audio-Visual Scene-Aware Dialog (AVSD), where AI agents must answer questions about videos without relying on human-generated captions at test time. The authors propose an extended Audio-Visual Transformer that uses Joint Student-Teacher Learning (JSTL) to mimic "privileged" information and a Region Proposal Network (RPN) to provide temporal evidence (reasoning) for its answers, achieving state-of-the-art results across multiple benchmarks.

What Problem Does This Paper Address?

The Audio-Visual Scene-Aware Dialog (AVSD) task requires an AI to watch a video, listen to its audio, and engage in a multi-turn conversation about the scene. In previous iterations of the AVSD challenge (DSTC7 and DSTC8), the top-performing systems relied heavily on human-written video descriptions. While these descriptions provide a massive performance boost, they aren't available in real-world scenarios—an AI assistant in your home won't have a pre-written script of what it's looking at.

Furthermore, most models act as "black boxes," providing an answer without explaining which part of the video led to that conclusion. This paper tackles two main gaps:

  1. Eliminating "Caption Dependency": Training models to perform well without human-generated summaries at inference time.
  2. Temporal Reasoning: Requiring the system to point to specific timestamps in the video as evidence for its answer.

Methods

The authors build upon a baseline AV-Transformer (an encoder-decoder architecture) and introduce several key enhancements:

  • Attentional Multimodal Fusion: Instead of simply concatenating audio and video features, they use Single-Head Attention (SHA) to dynamically weigh the importance of different modalities for each word prediction.
  • Joint Student-Teacher Learning (JSTL): This is the core innovation for handling the lack of captions. During training, a "Teacher" model has access to the video captions. A "Student" model (which lacks captions) is trained to minimize the difference between its output distribution and the Teacher's. This allows the Student to "learn" the descriptive nuances usually found in captions without needing them at test time.
  • Temporal Reasoning Modules: To provide evidence for answers, the authors compared two methods:
  • Attention-based: Using the model's internal attention weights to guess the relevant time frames.
  • RPN-based: A dedicated 1D-CNN Region Proposal Network that predicts the start, end, and confidence score of video segments relevant to the Question-Answer pair.
  • Model Combination: The final system ensembles the Transformer with a specialized LSTM-based model to further boost accuracy.

Datasets

The study utilizes the AVSD dataset, which is built on the Charades dataset (short, multi-action home videos).

  • DSTC7 & DSTC8: Used for benchmarking against previous state-of-the-art systems.
  • DSTC10-AVSD: A new extension introduced in this paper that includes human-annotated temporal reasoning data (specific video segments that support ground-truth answers).

Key Insights and Ideas

  • Captions as a Crutch: The authors highlight that previous models were essentially "reading" the answers from human captions rather than "seeing" them in the video. JSTL effectively transfers the "knowledge" of those captions into the visual/audio encoders.
  • Reasoning requires specialized architecture: Simply looking at attention weights (the "Attention-based method") is a poor way to find evidence. A dedicated RPN that treats evidence-finding as a detection task is significantly more effective.
  • Multimodal Synergy: The bimodal attention mechanism allows the model to find interdependencies (e.g., matching the sound of a door closing to the visual of a person entering a room), which is critical when text descriptions are missing.

Key Results

The proposed system achieved state-of-the-art (SOTA) performance on the DSTC7, DSTC8, and DSTC10 test sets:

  • Accuracy: On the DSTC10 test set, the extended AV-Transformer + LSTM ensemble improved the BLEU4 score from 0.247 (baseline) to 0.385 and the CIDEr score from 0.566 to 0.888.
  • Reasoning: The RPN-based reasoning method significantly outperformed the attention-based baseline, achieving an IoU-2 (Intersection over Union) of 0.550 vs. 0.380. This proves that training with explicit temporal annotations is vital for explainable AI in video tasks.

Limitations and Future Work

While the system achieves SOTA, the authors note that there is still a performance gap between models using manual descriptions and those that don't. Future work likely involves:

  • End-to-End Feature Learning: The current model uses pre-extracted features (VGGish and I3D); training the entire pipeline end-to-end might capture more granular details.
  • Complex Reasoning: Moving beyond single-segment evidence to multi-segment or causal reasoning (e.g., "Why did the person fall?").
  • Efficiency: Ensembling Transformers and LSTMs is computationally expensive, suggesting a need for more streamlined, single-model architectures that maintain this performance.

Summary updated

Abstract

In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.
BibTeX
@inproceedings{shah2022audio,
  author = {Ankit Shah and Shijie Geng and Peng Gao and Anoop Cherian and Takaaki Hori and Tim K. Marks and Jonathan Le Roux and Chiori Hori},
  title = {{Audio-Visual Scene-Aware Dialog and Reasoning Using Audio-Visual Transformers with Joint Student-Teacher Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=j-b7uDLIuC},
}

Loading related content...

Figure