1213Li at SemEval-2021 Task 6: Detection of Propaganda with Multi-modal Attention and Pre-trained Models

Peiguang Li, Xuan Li, Xian SunACL 2021

TL;DR

This paper tackles the challenge of automatically detecting propaganda techniques in memes, which combine text and images. The authors propose a novel multi-modal attention mechanism that effectively fuses features from both modalities, leading to a significant improvement over baseline methods and achieving a top-tier ranking in the SemEval-2021 competition.

What Problem Does This Paper Address?

The proliferation of online information, particularly through social media, has led to a surge in disinformation. Memes, a popular form of online content, are frequently used to spread propaganda due to their engaging nature and ability to convey messages through both text and images. Detecting these propaganda techniques is crucial for understanding and combating disinformation. SemEval-2021 Task 6, Subtask 3, specifically aimed to develop computational methods for identifying persuasion techniques in multi-modal meme content, which presents a unique challenge due to the interplay between textual and visual elements.

Methods

The proposed method by the 1213Li team is built upon a multi-layered architecture:

  • Extraction Layer: This layer leverages powerful pre-trained models for feature extraction. For text, RoBERTa is used to capture contextual information, with multiple sentences concatenated and separated by a newline character. For images, ResNet-50, pre-trained on ImageNet, extracts visual features.
  • Fusion Layer: This is the core of the novelty. To address both modality-specific and global propaganda techniques, the authors employ a two-pronged fusion strategy:
  • Modality-Specific Features: Average pooling is applied to both text and image features to obtain representations that capture the essence of each modality independently.
  • Global Features with Attention: A crucial innovation is the use of a multi-modal attention mechanism. This mechanism learns to align label embeddings with both textual and visual features. By calculating similarity scores, it weights the features based on their relevance to each specific propaganda label, allowing for a more nuanced understanding of how text and image interact to convey a particular technique. The final representation is a concatenation of these modality-specific and attention-weighted global features.
  • Classifier: A simple three-layer fully connected network with a sigmoid activation function is used to predict the presence of each propaganda label. This process is performed synchronously for all labels.

Datasets

The study utilizes the dataset provided for SemEval-2021 Task 6, Subtask 3. This dataset consists of image-text pairs, each labeled with one or more propaganda techniques. The dataset is divided into training, development, and test sets, with 687, 63, and 200 samples respectively.

Key Insights and Ideas

  • The Power of Multi-modal Attention: The paper strongly emphasizes that simply fusing features from text and images is insufficient. The attention mechanism, which learns to align features with specific labels, is key to understanding the complex interplay between modalities in conveying propaganda.
  • Label Embeddings Matter: Explicitly learning embeddings for each propaganda label and using them in the attention mechanism helps the model understand the semantic relationships between labels and how they are expressed in the multi-modal content.
  • Pre-trained Models are Essential: The success of the model hinges on the strong feature representations provided by pre-trained RoBERTa for text and ResNet-50 for images, demonstrating the effectiveness of transfer learning in this domain.
  • Global Techniques Require Deeper Fusion: The authors highlight that global techniques, which rely on the combined meaning of text and image, benefit significantly from sophisticated fusion methods like attention, rather than simple concatenation or averaging.

Key Results

The proposed method achieved significant improvements over the baseline, which relied on simple average pooling. The attention mechanism, in particular, led to substantial gains, especially in the Micro F1 score. The final model, incorporating both visual and textual features with the attention mechanism, achieved optimal performance. The team ranked 3rd out of 16 participants in the SemEval-2021 competition, with Micro F1 and Macro F1 scores of 0.54860 and 0.22830, respectively. An ablation study (Table 1) clearly demonstrates the incremental benefits of adding visual attention, textual attention, and finally, the full attention mechanism.

Limitations and Future Work

  • Limitations: The paper acknowledges that the baseline method, which only uses average-pooling, is "very ineffective," indicating that simpler fusion strategies are insufficient. While not explicitly stated as a limitation, the reliance on pre-trained models implies a dependency on the quality and scope of those models.
  • Future Work: The paper doesn't explicitly detail future work directions. However, the success of the attention mechanism suggests that further exploration of more sophisticated attention variants or cross-modal attention mechanisms could be beneficial. Additionally, investigating other pre-trained models or exploring different fusion strategies for specific types of propaganda techniques could be promising avenues. The ensemble of five models also hints at the potential for further performance gains through more advanced ensemble techniques.

Summary updated

BibTeX
@inproceedings{li2021li,
  author = {Peiguang Li and Xuan Li and Xian Sun},
  title = {{1213Li at SemEval-2021 Task 6: Detection of Propaganda with Multi-modal Attention and Pre-trained Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure