YNU-HPCC at SemEval-2021 Task 6: Combining ALBERT and Text-CNN for Persuasion Detection in Texts and Images

Xingyu Zhu, Jin Wang, Xuejie ZhangACL 2021

TL;DR

This paper tackles the challenge of automatically detecting persuasion techniques in memes, which are increasingly used for disinformation. The authors propose novel hybrid models combining powerful language models like ALBERT with convolutional neural networks (CNNs) for text analysis, and a parallel channel approach for multimodal (text + image) analysis, achieving competitive results in the SemEval-2021 Task 6.

What Problem Does This Paper Address?

Memes, with their potent blend of text and imagery, have become a ubiquitous tool in online communication and, unfortunately, a powerful vehicle for disinformation campaigns. Their effectiveness stems from their ability to convey messages quickly and emotionally, often employing subtle psychological and rhetorical tactics. Automatically identifying these persuasion techniques within memes is crucial for understanding and combating the spread of misinformation. SemEval-2021 Task 6 specifically aimed to address this by challenging participants to detect persuasion techniques in meme text and images.

Methods

The authors developed three distinct systems tailored to the three subtasks of SemEval-2021 Task 6:

  • Subtask 1 (Text-only Technique Detection): They proposed a hybrid model that fine-tunes the pre-trained ALBERT language model and then feeds its output into a Text-CNN layer. This combination aims to leverage ALBERT's deep contextual understanding of text while Text-CNN captures local linguistic patterns.
  • Subtask 2 (Text-only Technique Span Detection): For this more granular task, they adapted a BERT model for text feature extraction and then employed a multi-task sequence labeling approach. The problem was reframed as detecting each of the 20 persuasion techniques independently for each text segment.
  • Subtask 3 (Multimodal Technique Detection): This subtask required analyzing both text and images. The authors designed a parallel channel architecture. The text channel utilized their ALBERT-Text-CNN model, while the image channel employed pre-trained ResNet and VGGNet models for feature extraction. The features from both channels were then concatenated and fed into a classifier.

Datasets

The study utilized the dataset provided for SemEval-2021 Task 6. This dataset included:

  • Subtask 1: Meme IDs, textual content, and corresponding persuasion techniques.
  • Subtask 2: Meme IDs, textual content, and the specific text spans (start and end points) where persuasion techniques were applied.
  • Subtask 3: Meme IDs, textual content, and corresponding meme images, along with persuasion techniques applicable to either text or image.

The dataset consisted of 687 memes for training, 63 for development, and 200 for testing.

Key Insights and Ideas

  • Hybrid Models Shine: Combining powerful pre-trained language models like ALBERT with CNNs proved effective for text-based persuasion detection. This suggests that leveraging both deep contextual understanding and local feature extraction is key.
  • Multimodal Fusion is Crucial: For memes, analyzing only text is insufficient. The parallel channel approach for Subtask 3, effectively fusing text and image features, highlights the importance of a multimodal perspective.
  • ALBERT's Efficiency: The paper notes that ALBERT, being a "lite" BERT, demonstrated good performance, especially on smaller datasets, potentially due to its reduced parameter count and layer-to-layer parameter sharing. This makes it an attractive option when computational resources or dataset sizes are constraints.
  • Dev vs. Test Set Performance Gap: A notable observation is the significant drop in performance from the development set to the test set. This suggests potential challenges with generalization, possibly due to dataset imbalances or overfitting on the development data.

Key Results

The proposed systems achieved competitive results on the SemEval-2021 Task 6:

  • Subtask 1: The ALBERT-Text-CNN model achieved a micro F₁-score of 0.625 on the development set, outperforming other tested models like BERT and RoBERTa. On the test set, their micro F₁-score was 0.492, ranking 12th.
  • Subtask 2: The multi-task sequence labeling model achieved a micro F₁-score of 0.215 on the development set and 0.091 on the test set, ranking 7th.
  • Subtask 3: The parallel channel model (ALBERT-Text-CNN + ResNet18) achieved a micro F₁-score of 0.636 on the development set. On the test set, their micro F₁-score was 0.446, ranking 11th.

Across all subtasks, their results exceeded the baseline models.

Limitations and Future Work

The authors acknowledge a considerable decrease in performance from the development set to the test set. They attribute this to potential issues with unbalanced training data. For future work, they plan to focus on improving the generalization ability of their models by addressing these data imbalance problems.

Summary updated

BibTeX
@inproceedings{zhu2021ynu,
  author = {Xingyu Zhu and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2021 Task 6: Combining ALBERT and Text-CNN for Persuasion Detection in Texts and Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure