YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT

Chao Han, Jin Wang, Xuejie ZhangNAACL 2022

TL;DR

This paper presents a multi-modal, multi-label emotion classification model for identifying misogynous memes, leveraging the LXMERT architecture for cross-modal alignment and a novel bilinear-interactive fusion layer. The proposed model achieves state-of-the-art results on the SemEval-2022 Task 5, demonstrating the effectiveness of integrating visual and textual information for nuanced emotion detection in memes.

What Problem Does This Paper Address?

The proliferation of online content, particularly memes, has led to the need for automated systems capable of understanding their emotional and semantic nuances. Memes, often combining images and text, can convey complex emotions, including negative sentiments like misogyny. SemEval-2022 Task 5 specifically targets the identification of misogynous memes, posing two sub-tasks: a binary classification of whether a meme is misogynous or not (Sub-task A), and a multi-label classification to identify specific types of misogyny (e.g., stereotype, shaming, objectification, violence) (Sub-task B). Existing approaches often process modalities separately and then concatenate features, neglecting the crucial alignment between visual and textual elements. This paper aims to bridge this gap by proposing a model that effectively aligns and fuses multi-modal information for more accurate misogyny detection in memes.

Methods

The core of the proposed model is built upon the LXMERT (Cross-modal Encoder for Vision-Language Understanding) architecture. The authors adapt LXMERT for their multi-modal emotion classification task.

Here's a breakdown of their approach:

  • Image Feature Extraction: Instead of using raw image pixels, they employ Faster R-CNN (specifically, the bottom-up attention variant) to extract object-level visual features. This involves identifying 36 "region of interest" (RoI) boxes with high confidence within an image, along with their corresponding positional information. These RoI features are then embedded.
  • Text Feature Extraction: Standard BERT embeddings are used to represent the text associated with the meme. Word embeddings are combined with positional embeddings to capture the sequence of words.
  • Multi-modal Alignment: LXMERT's transformer-based encoder is utilized to process both the visual and textual features. This encoder includes self-attention sub-layers for each modality and cross-attention sub-layers that allow the model to learn relationships and alignments between the image and text representations. This is a key departure from simpler concatenation methods.
  • Modality Fusion: After LXMERT processes the aligned multi-modal features, a novel bilinear-interactive layer is introduced for fusion. This layer uses a trainable k-order square matrix to combine the image and text features, aiming for a more sophisticated integration than simple concatenation. This is followed by a dropout layer for regularization.
  • Output Layer:
    • For Sub-task A (binary classification), a single fully connected layer with a sigmoid activation function is used.
  • For Sub-task B (multi-label classification), a fully connected layer with 5 output units (corresponding to the 5 potential misogyny labels) and a sigmoid activation function is employed, treating each label as an independent binary classification problem.
  • Loss Function: To address the data imbalance observed in the dataset, the authors adopt Focal Loss. This loss function down-weights the contribution of easy-to-classify samples, allowing the model to focus more on difficult examples.

Datasets

The study utilizes the dataset provided for SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification (MAMI). This dataset consists of 10,000 meme images, each paired with corresponding text descriptions and labels for misogyny identification. The dataset is divided into training and testing sets. The authors note that the training dataset exhibits significant label imbalance, with some categories having much lower proportions than others.

Key Insights and Ideas

  • Importance of Cross-Modal Alignment: The paper strongly emphasizes that simply concatenating features from separate image and text models is insufficient. LXMERT's architecture, with its cross-attention mechanisms, is crucial for learning the intricate relationships between visual and textual content in memes, leading to better performance.
  • Bilinear-Interactive Fusion: The introduction of a bilinear-interactive layer for fusing features offers a more powerful mechanism for combining information from different modalities compared to standard concatenation. This suggests that higher-order interactions between features can capture richer semantic representations.
  • Focal Loss for Imbalanced Data: The use of Focal Loss is a practical and effective strategy for handling the inherent data imbalance in the MAMI dataset. This allows the model to learn more robustly from minority classes, which are often critical for accurate classification.
  • Object-Level Visual Features: Employing Faster R-CNN for object-level feature extraction, rather than global image features, allows the model to focus on salient objects within the meme that might be more relevant to conveying misogynistic sentiment.

Key Results

The proposed model, leveraging LXMERT with the bilinear-interactive fusion layer and Focal Loss, achieved the best results among the evaluated models.

  • Sub-task A (Misogynous Meme Identification): The model achieved an F1-score of 0.662.
  • Sub-task B (Specific Misogyny Type Identification): The model achieved an F1-score of 0.633.

These scores represent a significant improvement over baseline methods, including models that used only image or text, or combined ELECTRA and ResNet with simpler fusion techniques. The paper highlights that their model outperforms a baseline using LXMERT without the bilinear-interactive fusion layer, underscoring the effectiveness of their proposed fusion mechanism.

Limitations and Future Work

The authors acknowledge several limitations:

  • Object Detection Limitations: They observed that Faster R-CNN sometimes failed to detect relevant objects or characters in the meme images, potentially impacting accuracy. This could be due to the background of the meme or the image size being too small to contain multiple distinct targets.
  • Single-Modality Encoders: While LXMERT uses cross-modal encoders, the initial processing involves separate single-modality encoders. Future work could explore more deeply integrated multi-modal architectures from the outset.

The paper implicitly suggests future work could focus on improving object detection within memes and exploring more advanced multi-modal fusion techniques to further enhance performance.

Summary updated

BibTeX
@inproceedings{han2022ynu,
  author = {Chao Han and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2022 Task 5: Multi-Modal and Multi-label Emotion Classification Based on LXMERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure