Abstract
In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.
Introduction
The Audio-Visual Scene-Aware Dialog (AVSD) task requires an AI to watch a video, listen to its audio, and engage in a multi-turn conversation about the scene. While previous iterations of the AVSD challenge (DSTC7 and DSTC8) saw significant progress, top-performing systems relied heavily on human-written video descriptions. These descriptions act as a "crutch" that would not be available in real-world applications, such as a home AI assistant.
This research addresses two critical gaps in the field:
- Eliminating Caption Dependency: Training models to perform effectively without human-generated summaries at inference time.
- Temporal Reasoning: Requiring systems to provide explainable evidence by pointing to specific timestamps in the video that support their answers.
Methodology
The proposed system builds upon an encoder-decoder AV-Transformer architecture, enhanced with several key innovations to handle multimodal inputs and the absence of captions at test time.
Attentional Multimodal Fusion
Instead of simple concatenation, the model employs Single-Head Attention (SHA) to dynamically weight the importance of audio and visual modalities for each word prediction. This allows the model to focus on the most relevant signals (e.g., a specific sound or a visual action) during the dialog.
Joint Student-Teacher Learning (JSTL)
To overcome the lack of captions at inference, the authors introduce Joint Student-Teacher Learning (JSTL). During training, a "Teacher" model has access to privileged information (video captions), while a "Student" model does not. The Student is trained to minimize the difference between its output distribution and the Teacher's, effectively transferring the descriptive knowledge usually found in captions into the visual and audio encoders.
Temporal Reasoning Modules
To provide evidence for its answers, the system uses a Region Proposal Network (RPN). This 1D-CNN predicts the start and end times, along with a confidence score, for video segments relevant to the question-answer pair. This was found to be significantly more effective than simply using internal attention weights.
Experimental Setup
The study utilizes the AVSD dataset, built on the Charades collection of short home videos. For the DSTC10 challenge, the authors introduced a new extension featuring human-annotated temporal reasoning data, where specific video segments are linked to ground-truth answers.
Results and Discussion
The extended AV-Transformer, combined with an LSTM ensemble, achieved state-of-the-art (SOTA) performance across DSTC7, DSTC8, and DSTC10 benchmarks. On the DSTC10 test set, the system improved the BLEU4 score from a baseline of 0.247 to 0.385 and the CIDEr score from 0.566 to 0.888.
A key finding was the superiority of the RPN-based reasoning method. As shown in the table below, the RPN significantly outperformed the attention-based baseline in terms of Intersection over Union (IoU) metrics.
These results demonstrate that explicit temporal annotations and specialized detection architectures are vital for creating explainable AI in video-based dialog tasks.
Conclusion
This work advances the AVSD task toward real-world applicability by removing the dependency on human captions and introducing a robust framework for temporal reasoning. The combination of JSTL for knowledge transfer and RPN for evidence detection sets a new standard for scene-aware dialog systems. Future work will explore end-to-end feature learning and more complex causal reasoning.
Summary updated