DUTIR938 at SemEval-2024 Task 4: Semi-Supervised Learning and Model Ensemble for Persuasion Techniques Detection in Memes

Erchen Yu, Junlong Wang, Xuening Qiao, Jiewei Qi, Zhaoqing Li, Hongfei Lin, Linlin Zong, Bo XuNAACL 2024

TL;DR

This paper tackles the challenge of automatically detecting persuasion techniques in memes, a growing vector for online disinformation. The authors propose a novel dual-channel model that combines image and text features, enhanced by semi-supervised learning and a sophisticated ensemble strategy, achieving top performance in the SemEval-2024 Task 4.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to a surge in online disinformation. Memes, with their potent blend of images and text, have emerged as a particularly effective medium for spreading propaganda and influencing public opinion, often without users consciously realizing it. Detecting the subtle persuasion techniques embedded within these memes is crucial for understanding user-generated content and combating the spread of misinformation. However, the multimodal nature of memes and the inherent subjectivity of persuasive language make this a complex task for automated systems.

Methods

The DUTIR938 team developed a dual-channel model that processes both image and text modalities separately before fusing them.

  • Image Feature Extraction: They utilized CLIP (Contrastive Language-Image Pre-training) to extract rich visual features from meme images.
  • Text Feature Extraction: For text, they experimented with various Transformer-based language models, including BERT, RoBERTa, and DeBERTa, and specifically selected models pre-trained on political domain data (politicalBiasBERT, xlm-twitter-politics-sentiment) due to the nature of persuasion techniques. Task-adaptive pretraining (TAPT) was applied to these text models using data from the task to further improve their relevance.
  • Cross-Modal Fusion: The extracted image and text features were concatenated and fed into a three-layer fully connected network for classification.
  • Semi-Supervised Learning (Pseudo-labeling): To augment the limited labeled data, they employed a pseudo-labeling strategy. Models were trained on the labeled data, then used to predict labels for unlabeled test samples. High-confidence predictions were added to the training set, effectively expanding the dataset.
  • Adversarial Training (FGM): To enhance model robustness, they incorporated Fast Gradient Method (FGM) to introduce small perturbations to the model's embedding layer, making it more resilient to variations.
  • Model Ensemble: A two-stage soft-voting ensemble strategy was used. First, k-fold cross-validation was applied to train multiple models. In the first stage of the ensemble, probabilities from models within each fold were averaged. In the second stage, probabilities from all folds were averaged to produce the final prediction.

Datasets

The primary dataset used was provided by SemEval-2024 Task 4, Subtask 2b. The original dataset comprised 1200 training samples, 150 validation samples, 300 development samples, and 600 test samples. After pseudo-labeling, the test set was augmented with an additional 101 non-propagandistic and 292 propagandistic samples. The authors also mention using task-specific corpora for task-adaptive pretraining.

Key Insights and Ideas

  • Multimodality is Key: Effectively combining image and text features is essential for understanding the persuasive intent of memes.
  • Domain-Specific Pretraining Matters: Leveraging language models pre-trained on political text significantly boosted performance, highlighting the importance of domain relevance.
  • Semi-Supervised Learning is Crucial for Small Datasets: Pseudo-labeling proved effective in overcoming data scarcity, a common challenge in specialized NLP tasks.
  • Robustness Enhancements Pay Off: Both adversarial training (FGM) and task-adaptive pretraining (TAPT) demonstrably improved model performance.
  • Ensembling for Generalization: The multi-stage ensemble strategy was vital for achieving top-tier results, showcasing its ability to consolidate the strengths of individual models.

Key Results

The proposed system achieved strong results on the English leaderboard for SemEval-2024 Task 4, Subtask 2b:

  • Rankings: The team ranked 3rd out of 19 teams in terms of Macro F1 and 1st out of 19 teams in terms of Micro F1.
  • Performance Metrics: The final submission achieved a Macro F1 score of 0.80910 and a Micro F1 score of 0.83667.
  • Effectiveness of Strategies: Table 3 shows that the combination of FGM and TAPT with base models led to significant performance improvements, with the politicalBiasBERT model combined with FGM and TAPT achieving the highest Macro F1 score of 0.8586 on their validation set. The all-model ensemble strategy also demonstrated superior performance compared to individual models and smaller ensembles.

Limitations and Future Work

  • Limitations: The paper acknowledges the inherent complexity and subjectivity of persuasive language, which can still pose challenges. The relatively small size of the dataset, even after pseudo-labeling, is a factor that the ensemble and semi-supervised methods were designed to mitigate.
  • Future Work: The authors suggest that future research will focus on exploring more advanced cross-modal fusion mechanisms within the model to further enhance the integration of image and text information.

Summary updated

BibTeX
@inproceedings{yu2024dutir,
  author = {Erchen Yu and Junlong Wang and Xuening Qiao and Jiewei Qi and Zhaoqing Li and Hongfei Lin and Linlin Zong and Bo Xu},
  title = {{DUTIR938 at SemEval-2024 Task 4: Semi-Supervised Learning and Model Ensemble for Persuasion Techniques Detection in Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure