Adversarial Multimodal Network for Movie Question Answering

Lixin DuanarXiv 2019 · arXiv

TL;DR

This paper introduces the Adversarial Multimodal Network (AMN) for movie question answering, a task that requires understanding both visual scenes and textual subtitles. AMN leverages a Generative Adversarial Network (GAN) framework to learn more coherent multimodal representations and employs a self-attention mechanism to preserve crucial story cues, achieving state-of-the-art results on the MovieQA dataset.

What Problem Does This Paper Address?

Visual Question Answering (VQA) has seen significant advancements, but extending it to understand complex narratives in movies, known as multimodal movie question answering (MovieQA), presents unique challenges. Movies combine visual information with rich textual content like subtitles, but these modalities have vastly different statistical properties, making it difficult to bridge the gap. Furthermore, subtitles and visual clips are often not perfectly aligned in time, and the core story elements can be scattered across a long movie. Existing methods often focus on aligning visual features to textual space but overlook the distribution of learned multimodal representations and the critical issue of retaining story information. This paper aims to address this by learning robust multimodal representations that are both coherent across modalities and preserve essential narrative cues.

Methods

The core of the proposed method is the Adversarial Multimodal Network (AMN), which frames multimodal representation learning within a GAN architecture.

  • Generator: The generator's role is to map visual clip features and subtitle features into a shared multimodal representation space. It employs a two-layered attention mechanism:
  • The first layer maps regional visual features to a word dictionary.
  • The second layer attends to embedded subtitle features. This process aims to create representations that are indistinguishable from the corresponding textual data (subtitles and questions).

  • Discriminator: The discriminator's task is to distinguish between the generated multimodal representations and the "real" textual data (subtitles and questions). This adversarial process pushes the generator to produce more realistic and coherent multimodal representations.

  • Self-Attention based Consistency Constraints: To combat the potential loss of information during the projection process, especially for story cues, a self-attention mechanism is introduced. This mechanism maps both visual representations and the generated multimodal representations into a fixed-dimensional space. By enforcing a consistency constraint (measured by cosine distance) between these self-attention outputs, the model ensures that the intrinsic relationships and long-term dependencies within the movie's story are preserved.
  • Answer Inference: The learned multimodal representations are then used to predict the answer. The method incorporates an update step for subtitle features using the question, answer, and visual context, aiming to filter out irrelevant information and focus on pertinent cues.

Datasets

The experiments were conducted on the MovieQA dataset. This dataset comprises multiple-choice questions about movies, with information sources including video clips, plots, subtitles, scripts, and DVS transcriptions. For this work, the "Video+Subtitles" task was focused on, resulting in 6,462 question-answer pairs from 140 movies.

Key Insights and Ideas

  • GANs for Multimodal Coherence: Applying GANs to multimodal representation learning is a novel approach. The adversarial nature encourages the generator to produce representations that are not only informative but also align well with the textual domain, bridging the modality gap.
  • Self-Attention for Story Preservation: The self-attention mechanism is crucial for retaining "story cues." This is particularly important in long narratives where key information might be spread out, and simple projection might lose these temporal dependencies.
  • Complementary Components: The ablation study highlights that both the adversarial learning and the self-attention consistency constraints are independently beneficial and work synergistically to improve performance.
  • Improved Generalization: The AMN shows consistent performance on both validation and test sets, suggesting good generalization capabilities, which is a significant challenge in multimodal tasks.
  • Feature Distribution Alignment: Visualizations demonstrate that AMN's learned feature distributions for subtitles and questions show a much larger overlap compared to previous methods like LMN, indicating a more effective integration of modalities.

Key Results

  • State-of-the-Art Performance: The proposed AMN achieved state-of-the-art accuracy on the MovieQA dataset, outperforming existing methods like FVTA and LMN. The reported accuracy on the test set was 45.31%.
  • Ablation Study: Both the adversarial component (AMNadv) and the self-attention consistency component (AMNcons) individually improved performance over a baseline (AMNdeg, similar to LMN), with their combination (AMN) yielding the best results.
  • Qualitative Improvements: Qualitative examples show that AMN can correctly predict answers where LMN fails, demonstrating its ability to associate subtitles with more relevant frames and find better correspondences for question-answer pairs.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of the MovieQA task itself. However, the reliance on pre-trained VGG-16 for visual features and word2vec for text embeddings suggests potential limitations if these pre-trained models are not optimal for the specific movie domain.
  • Future Work: The paper's conclusion suggests that the AMN framework is a promising direction for multimodal question answering. Future work could explore more advanced GAN architectures, different attention mechanisms, or investigate how to better handle the temporal dynamics and potential misalignments in video and subtitle data. The authors also mention that their method outperforms existing baselines, implying that further research could build upon this framework to explore even more complex multimodal reasoning tasks.

Summary updated

BibTeX
@misc{duan2019adversarial,
  author = {Lixin Duan},
  title = {{Adversarial Multimodal Network for Movie Question Answering}},
  year = {2019},
  eprint = {1906.09844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.09844},
}
Figure