CSECU-DSG at SemEval-2021 Task 6: Orchestrating Multimodal Neural Architectures for Identifying Persuasion Techniques in Texts and Images

Tashin Hossain, Jannatun Naim, Fareen Tasneem, Radiathun Tasnia, Abu Nowshed ChyACL 2021

TL;DR

This paper presents CSECU-DSG's participation in SemEval-2021 Task 6, focusing on identifying persuasion techniques in text and images. They developed a multimodal approach combining various neural architectures, including BERT variants, CNNs, LSTMs, and ResNets, to achieve competitive results across three subtasks.

What Problem Does This Paper Address?

The paper tackles the critical challenge of detecting persuasion techniques, particularly in multimodal contexts like memes. These techniques are increasingly used to subtly influence public opinion, spread propaganda, and even disseminate fake news. The SemEval-2021 Task 6 aimed to develop systems capable of identifying these techniques in textual content, pinpointing their exact location within text, and recognizing them in multimodal (text + image) content. This is crucial for protecting users from manipulation and deception.

Methods

The CSECU-DSG team employed a multi-pronged strategy across the three subtasks:

  • Subtask 1 (Textual Persuasion Technique Classification): They fused the predictions of three models:
  • Logistic Regression: Used Tf-Idf vectorization for feature representation.
  • Decision Tree: A supervised learning classifier for splitting data based on parameters.
  • Fine-tuned DistilBERT: A smaller, faster version of BERT, fine-tuned on the task's dataset. A majority voting scheme was used to combine the outputs of these models.

  • Subtask 2 (Span Identification of Persuasive Techniques): This involved a two-stage process:

  • Span Identification: A pre-trained BERT model was used to classify phrases within text as "Persuasive" or "Not Persuasive." This leveraged data from SemEval 2020 Task 11.
  • Technique Classification: The identified persuasive phrases were then fed into another pre-trained BERT model for multi-label classification of specific persuasion techniques.
  • Subtask 3 (Multimodal Multi-label Classification): This subtask combined text and image information. Their approach involved a majority voting fusion of three models:
  • Fine-tuned DistilBERT: For text analysis.
  • Fine-tuned ResNet50: A deep convolutional neural network for image analysis.
  • Ensemble of Multi-kernel CNN + LSTM and MLP: A more complex architecture designed to capture both local and sequential features from text, and also incorporating image features.

Datasets

The primary dataset used was from SemEval-2021 Task 6, which provided data for all three subtasks. For Subtask 2, they also utilized the SemEval-2020 Task 11 dataset (both for span identification and technique classification) to augment their training data and improve performance. The iMet Collection 2019 - FGVC6 dataset was used for fine-tuning the ResNet50 model in Subtask 3.

Key Insights and Ideas

  • Fusion for Robustness: The consistent use of majority voting to fuse predictions from multiple diverse models (e.g., traditional ML with deep learning, or different deep learning architectures) proved effective in improving overall performance across subtasks. This highlights the benefit of ensemble methods in complex NLP and multimodal tasks.
  • Transfer Learning is Key: The paper heavily relies on pre-trained transformer models like BERT and DistilBERT, and pre-trained image models like ResNet50. This demonstrates the power of transfer learning in achieving strong results on specialized tasks with limited task-specific data.
  • Multimodality is Challenging but Rewarding: Subtask 3, dealing with both text and images, presented a significant challenge. The authors' approach of combining specialized text and image models, then fusing their outputs, is a standard yet effective strategy for multimodal tasks.
  • Label Imbalance is a Hurdle: The authors explicitly mention that label imbalance in the dataset led to difficulties in detecting less frequent persuasion techniques. This is a common problem in real-world classification tasks and suggests a need for more sophisticated handling of imbalanced data in future work.

Key Results

The CSECU-DSG system achieved competitive performance across all three subtasks, as evidenced by their F1-Micro scores. While not always the top-performing team, their systems consistently ranked well.

  • Subtask 1: Their fused model achieved an F1-Macro score of 0.48894. The individual DistilBERT model performed best among their three models for this subtask, achieving 0.48064.
  • Subtask 2: Their system achieved an F1-Score of 0.11983.
  • Subtask 3: Their fused model achieved an F1-Micro score of 0.51312. The individual fine-tuned DistilBERT model for text achieved 0.52899, slightly outperforming the majority voting model in this specific instance, but the authors still favored the multimodal fusion for the overall task.

Limitations and Future Work

  • Label Imbalance: The authors acknowledge that the imbalance of persuasion techniques in the dataset hindered the detection of less frequent labels.
  • Future Work:
    • For Subtasks 1 and 2, they plan to explore more advanced pre-processing techniques and experiment with more efficient classifiers and other transfer learning models like RoBERTa and GPT.
  • For Subtask 3, they aim to incorporate a wider variety of image datasets to extract more effective features.

Summary updated

BibTeX
@inproceedings{hossain2021csecu,
  author = {Tashin Hossain and Jannatun Naim and Fareen Tasneem and Radiathun Tasnia and Abu Nowshed Chy},
  title = {{CSECU-DSG at SemEval-2021 Task 6: Orchestrating Multimodal Neural Architectures for Identifying Persuasion Techniques in Texts and Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure