Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation

Kuan-Yen Lin, Chao-Chun Hsu, Yun-Nung Chen, Lun-Wei KuarXiv 2019 · OpenReview

TL;DR

This paper introduces an "Entropy-Enhanced Dynamic Memory Network" (DMN) to improve video understanding for scene-aware dialogue generation. By incorporating an entropy mechanism into the DMN, the model can more effectively focus on relevant video segments, leading to better dialogue responses that leverage multimodal information.

What Problem Does This Paper Address?

The proliferation of online videos has created a need for systems that can understand and reason about their content. While visual question answering (VQA) has made strides with static images, extending this to dynamic videos, especially in the context of conversations, is a significant challenge. The Audio Visual Scene-aware Dialog (AVSD) task, introduced in DSTC7, aims to address this by requiring models to generate dialogue responses based on a video, its dialogue history, summary, and caption. The core problem is effectively modeling the rich, sequential, and often noisy information present in videos to generate contextually relevant and accurate dialogue.

Methods

The authors propose an Entropy-Enhanced Dynamic Memory Network (DMN) to tackle the video modality. The DMN, known for its ability to store and retrieve sequential information, is adapted here. Key components of their approach include:

  • Dynamic Memory Network (DMN): This framework is used to process video features. It consists of an input module, an episodic memory module, and an answer module. The episodic memory module is crucial for retaining information over time.
  • Attention-based GRU: Within the DMN's episodic memory, an attention-based GRU is employed. This allows the model to leverage positional information within the video and focus on relevant segments guided by the question.
  • Entropy Enhancement: A novel contribution is the addition of an entropy term to the DMN's loss function. This mechanism encourages a sparser attention distribution, forcing the model to concentrate its focus on a smaller, more relevant set of video segments for each question. This helps to denoise the video information.
  • Multimodal Fusion: The model integrates features from multiple modalities:
  • Video: Processed by the entropy-enhanced DMN for both audio and visual streams.
  • Dialogue History: Encoded using a Bidirectional LSTM.
  • Caption and Summary: Processed using a basic attention mechanism.
  • These fused representations are then used by a decoder to generate the final dialogue response.

Datasets

The primary dataset used is from the Audio Visual Scene-aware Dialog (AVSD) task of the Dialog System Technology Challenge (DSTC7). This dataset includes:

  • Videos
  • Dialogue history (previous QA pairs)
  • Captions (descriptions of videos)
  • Summaries (conclusions about dialogues)
  • Questions about video content

The dataset consists of 7,659 training samples, 1,787 validation samples, and 1,710 testing samples.

Key Insights and Ideas

  • Video as a Noisy but Crucial Modality: While text-based modalities like captions and summaries are less noisy, videos contain essential scene-aware information that text alone cannot fully capture. The challenge lies in effectively extracting this information.
  • DMN for Video Memory: Dynamic Memory Networks are well-suited for video because their episodic memory allows for better recall of sequential information, which is fundamental to understanding video content.
  • Entropy for Focused Attention: The sparsity induced by the entropy mechanism is key. It helps the model avoid being overwhelmed by the vast amount of information in a video and instead directs its attention to the most pertinent parts for answering a specific question.
  • Dialogue History as Data Augmentation: Including dialogue history significantly improves performance, highlighting the importance of conversational context in scene-aware dialogue generation.
  • Textual Modalities Complement Video: While video is crucial, captions and summaries also play a vital role in improving response generation, suggesting a synergistic relationship between modalities.

Key Results

The proposed Entropy-Enhanced DMN system achieved improved performance over the released baseline model on the DSTC7 AVSD task.

  • Objective Metrics: Table 1 shows that the Entropy-Enhanced DMN (A, V, D, S, C) consistently outperformed the baseline across various metrics like BLEU, METEOR, ROUGE-L, and CIDEr. For instance, the CIDEr score reached 0.843, compared to the baseline's 0.766.
  • Subjective Metrics: Table 2, representing the official evaluation, shows the Entropy-Enhanced DMN (A, V, D, S, C) achieving a Human score of 3.048, significantly higher than the baseline's 2.848.
  • Ablation Studies: Experiments demonstrated that excluding video significantly degraded performance. They also showed that while audio features are less noisy than visual features, both are necessary for optimal results. The number of episodic memories (M=2) was found to be a good balance between performance and computational cost.

Limitations and Future Work

  • Training Data Quality: The authors note that the ground truth answers in the dataset can sometimes be ambiguous or generic, potentially hindering model learning. They also point out instances where captions and summaries are unhelpful for answering specific questions.
  • Model Complexity: Increasing the number of episodic memories (e.g., to M=3) showed only marginal improvements while drastically increasing training and testing time.
  • Future Work: The paper implicitly suggests that improving the quality and annotation of training data could lead to further gains. Additionally, exploring more sophisticated multimodal fusion techniques or attention mechanisms could be avenues for future research. The current work focuses on improving video understanding within the DMN framework, leaving room for advancements in other components of the dialogue generation pipeline.

Summary updated

Abstract

With increasing information from social media, there are more and more videos available. Therefore, the ability to reason on a video is important and deserves to be discussed. TheDialog System Technology Challenge (DSTC7) (Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task, which contains five modalities including video, dialogue history, summary, and caption, as a scene-aware environment. In this paper, we propose the entropy-enhanced dynamic memory network (DMN) to effectively model video modality. The attention-based GRU in the proposed model can improve the model's ability to comprehend and memorize sequential information. The entropy mechanism can control the attention distribution higher, so each to-be-answered question can focus more specifically on a small set of video segments. After the entropy-enhanced DMN secures the video context, we apply an attention model that in-corporates summary and caption to generate an accurate answer given the question about the video. In the official evaluation, our system can achieve improved performance against the released baseline model for both subjective and objective evaluation metrics.
BibTeX
@misc{lin2019entropy,
  author = {Kuan-Yen Lin and Chao-Chun Hsu and Yun-Nung Chen and Lun-Wei Ku},
  title = {{Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation}},
  year = {2019},
  eprint = {1908.08191},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.08191},
}
Figure