Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning

Ankit Shah, Shijie Geng, Peng Gao, Anoop Cherian, Takaaki Hori, Tim K. Marks, Jonathan Le Roux, Chiori Hori

Abstract

In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.

Introduction

The Audio-Visual Scene-Aware Dialog (AVSD) task requires an AI to watch a video, listen to its audio, and engage in a multi-turn conversation about the scene. While previous iterations of the AVSD challenge (DSTC7 and DSTC8) saw significant progress, top-performing systems relied heavily on human-written video descriptions. These descriptions act as a "crutch" that would not be available in real-world applications, such as a home AI assistant.

This research addresses two critical gaps in the field:

  1. Eliminating Caption Dependency: Training models to perform effectively without human-generated summaries at inference time.
  2. Temporal Reasoning: Requiring systems to provide explainable evidence by pointing to specific timestamps in the video that support their answers.

Methodology

The proposed system builds upon an encoder-decoder AV-Transformer architecture, enhanced with several key innovations to handle multimodal inputs and the absence of captions at test time.

Attentional Multimodal Fusion

Instead of simple concatenation, the model employs Single-Head Attention (SHA) to dynamically weight the importance of audio and visual modalities for each word prediction. This allows the model to focus on the most relevant signals (e.g., a specific sound or a visual action) during the dialog.

Joint Student-Teacher Learning (JSTL)

To overcome the lack of captions at inference, the authors introduce Joint Student-Teacher Learning (JSTL). During training, a "Teacher" model has access to privileged information (video captions), while a "Student" model does not. The Student is trained to minimize the difference between its output distribution and the Teacher's, effectively transferring the descriptive knowledge usually found in captions into the visual and audio encoders.

Architecture of the proposed AV-transformer with JSTL.

Temporal Reasoning Modules

To provide evidence for its answers, the system uses a Region Proposal Network (RPN). This 1D-CNN predicts the start and end times, along with a confidence score, for video segments relevant to the question-answer pair. This was found to be significantly more effective than simply using internal attention weights.

Experimental Setup

The study utilizes the AVSD dataset, built on the Charades collection of short home videos. For the DSTC10 challenge, the authors introduced a new extension featuring human-annotated temporal reasoning data, where specific video segments are linked to ground-truth answers.

Results and Discussion

The extended AV-Transformer, combined with an LSTM ensemble, achieved state-of-the-art (SOTA) performance across DSTC7, DSTC8, and DSTC10 benchmarks. On the DSTC10 test set, the system improved the BLEU4 score from a baseline of 0.247 to 0.385 and the CIDEr score from 0.566 to 0.888.

A key finding was the superiority of the RPN-based reasoning method. As shown in the table below, the RPN significantly outperformed the attention-based baseline in terms of Intersection over Union (IoU) metrics.

Temporal reasoning performance: Attention vs. RPN.

These results demonstrate that explicit temporal annotations and specialized detection architectures are vital for creating explainable AI in video-based dialog tasks.

Conclusion

This work advances the AVSD task toward real-world applicability by removing the dependency on human captions and introducing a robust framework for temporal reasoning. The combination of JSTL for knowledge transfer and RPN for evidence detection sets a new standard for scene-aware dialog systems. Future work will explore end-to-end feature learning and more complex causal reasoning.

Summary updated

Abstract

In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.
BibTeX
@inproceedings{shah2022audio,
  author = {Ankit Shah and Shijie Geng and Peng Gao and Anoop Cherian and Takaaki Hori and Tim K. Marks and Jonathan Le Roux and Chiori Hori},
  title = {{Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=j-b7uDLIuC},
}
Figure