Multimodal Weighted Fusion of Transformers for Movie Genre Classification

Isaac Rodríguez Bribiesca, Adrián Pastor López Monroy, Manuel Montes-y-GómezNAACL 2021

TL;DR

This paper tackles the challenge of effectively fusing multiple data sources (text, video, audio, etc.) for movie genre classification. They propose a novel adaptation of the Multimodal Transformer (MulT) architecture, replacing its standard concatenation-based late fusion with a Gated Multimodal Unit (GMU) to dynamically weight the importance of each modality per movie, leading to improved performance and interpretability.

What Problem Does This Paper Address?

The proliferation of online content, especially video platforms, has led to an explosion of multimodal data. Understanding this data automatically is crucial, and movie genre classification is a prime example. While Transformer-based models like MulT have shown promise in multimodal tasks, their performance can degrade when dealing with a large number of modalities. A key issue is how these modalities are fused. Traditional late fusion methods, like simple concatenation, can become a bottleneck, hindering the model's ability to learn nuanced representations. Furthermore, understanding why a model makes a certain prediction becomes difficult when relying solely on attention matrices. This paper aims to address these limitations by proposing a more intelligent and interpretable fusion mechanism.

Methods

The core of this work is an extension of the Multimodal Transformer (MulT) architecture. MulT, originally designed for human multimodal language understanding, uses cross-modal attention to fuse pairs of modalities (e.g., text and video) before a final late fusion step. The authors' key innovation is replacing MulT's default late fusion by concatenation with a Gated Multimodal Unit (GMU).

Here's how the GMU works:

  • Dynamic Weighting: For each input movie, the GMU learns a set of "gates" that dynamically determine the contribution of each modality to the final fused representation. This allows the model to prioritize modalities that are more relevant for a specific movie's genre (e.g., visual style for animation, audio for horror).
  • Interpretability: The learned gates provide insights into which modalities the model considers important for classifying a particular movie, offering a more interpretable view than raw attention maps.
  • Architecture: The MulT model first processes individual modalities and then uses cross-modal transformers to learn inter-modal relationships. The outputs from these cross-modal transformers, representing each modality's refined features, are then fed into the GMU for the final weighted fusion before prediction.

Datasets

The experiments were conducted on the Moviescope dataset. This is a large-scale dataset containing approximately 5,000 movies, with associated data including:

  • Movie trailers (video and audio)
  • Movie posters (images)
  • Movie plots (text)
  • Metadata

The dataset is pre-processed, with specific feature extraction methods detailed for each modality (e.g., BERT for text, CNN features for video and posters).

Key Insights and Ideas

  • Modality Relevance Varies by Genre: The paper strongly suggests that the importance of different modalities for genre classification is not uniform. For instance, text is generally the most informative, while audio is often the least. However, specific genres might leverage certain modalities more heavily (e.g., visual style for animation, audio cues for horror).
  • GMU for Smarter Fusion: Replacing simple concatenation with a GMU significantly improves performance by allowing the model to learn instance-specific modality weights. This is a crucial step beyond just having multiple modalities available.
  • Enhanced Interpretability: The GMU's gating mechanism offers a more direct way to understand modality importance compared to analyzing complex attention matrices, making the model's decisions more transparent.
  • Transformer Power Extended: The work demonstrates the adaptability and effectiveness of Transformer architectures, like MulT, for complex multimodal tasks beyond their initial NLP focus.

Key Results

The proposed MulT-GMU model consistently outperformed both the original MulT-Concat (using concatenation for late fusion) and the Fast-MA (a state-of-the-art baseline) on the movie genre classification task.

  • Significant Improvements: MulT-GMU achieved performance gains of 4%-10% across various metrics (μAP, mAP, sAP) when multiple modalities were considered.
  • Handling Difficult Modalities: The GMU was particularly effective in scenarios where text was excluded, showing better performance than MulT-Concat in handling metadata features, which often posed a challenge for the latter.
  • Text is King (Mostly): The results reinforce previous findings that text is a highly influential modality for movie genre classification. When text was included, MulT models showed dramatic performance increases.

Limitations and Future Work

  • Limitations: The paper acknowledges that while MulT-GMU improves interpretability, it's still a complex deep learning model. The qualitative analysis provides insights, but a more comprehensive interpretability framework could be beneficial. The performance drop in settings without text for MulT models, while explained by the importance of text, still highlights a challenge in effectively leveraging non-textual modalities in isolation.
  • Future Work: The authors suggest exploring further adaptations of the GMU module and investigating its application to other multimodal tasks. They also implicitly point towards the need for more robust fusion strategies that can better handle scenarios where certain modalities are less informative. The paper also marks the first time multimodal transformers are tested for movie genre classification, opening avenues for further research in this specific domain.

Summary updated

BibTeX
@inproceedings{bribiesca2021multimodal,
  author = {Isaac Rodríguez Bribiesca and Adrián Pastor López Monroy and Manuel Montes-y-Gómez},
  title = {{Multimodal Weighted Fusion of Transformers for Movie Genre Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure