Puer at SemEval-2024 Task 4: Fine-tuning Pre-trained Language Models for Meme Persuasion Technique Detection

Jiaxu Dao, Zhuoying Li, Youbang Su, Wensheng GongNAACL 2024

TL;DR

This paper tackles the challenge of automatically detecting persuasion techniques in memes, a crucial task for understanding online discourse and combating misinformation. The authors fine-tuned a pre-trained RoBERTa model on a social media meme dataset, achieving a 7th place ranking in the SemEval-2024 Task 4 for English meme persuasion technique detection.

What Problem Does This Paper Address?

Memes have become a dominant form of communication on social media, leveraging psychological manipulation and blurred logic to spread messages rapidly. Understanding the persuasive techniques employed in memes is vital for various applications, such as identifying hate speech, racial discrimination, deceptive information, and even for market trend analysis and political communication. However, automatically classifying these techniques, especially in a multimodal format like memes (combining images and text), is a complex challenge. This paper addresses the need for robust models capable of accurately identifying persuasion techniques within memes, specifically focusing on the SemEval-2024 Task 4, which involves a multi-label classification of these techniques.

Methods

The core of the proposed method involves fine-tuning a pre-trained language model. Specifically, the authors used the roberta-large model, which was further fine-tuned on a corpus of social media posts. This fine-tuning process adapted the general language understanding capabilities of RoBERTa to the specific nuances of meme language and persuasion techniques. The model construction involved standard steps: input processing using a tokenizer, a dropout layer for regularization, a fully connected layer for feature transformation, and a Binary Cross Entropy with Logits Loss function for multi-label classification. The output layer was designed to classify 20 distinct persuasion techniques plus a "none" category.

Datasets

The experiments utilized four distinct datasets: training, validation, development (dev), and test sets, all formatted in JSON. The training set contained 7,000 entries, the validation set 500, the development set 1,000, and the test set 1,500 instances. The paper highlights that the dataset features a minimum sentence length of one, with average sentence lengths varying across the sets. The "Smear" category was noted as the most frequent, while "Obfuscation," "Intentional vagueness," and "Confusion" were the least frequent. The authors also mention that their work is based on a roberta-large model pre-trained on social media posts, which was fine-tuned on a corpus of tweets and Instagram posts, including captions and comments.

Key Insights and Ideas

  • Fine-tuning RoBERTa for Memes is Effective: The study demonstrates that fine-tuning a powerful pre-trained language model like RoBERTa on a specific dataset of social media memes significantly improves performance in detecting persuasion techniques.
  • Social Media Pre-training Matters: The choice of a roberta-large model pre-trained on social media data (tweets and Instagram posts) proved beneficial, suggesting that domain-specific pre-training is crucial for capturing the language style and context of memes.
  • Threshold Selection is Critical: The paper emphasizes the importance of carefully selecting a threshold for classification. They found that a threshold of approximately 0.08 yielded the most optimal hierarchical F1 score on the development set, balancing precision and recall.
  • Epoch Selection Impacts Performance: The study observed that performance metrics like precision and hierarchical F1 scores stabilize and improve as the number of training epochs increases, with a notable stabilization around 20 epochs.

Key Results

The fine-tuned roberta-large model achieved a hierarchical F1 score of 0.647 on the development set and 0.66 on the test set for English-Subtask 1. This performance secured the team 7th place in the SemEval-2024 Task 4 leaderboard. The model also achieved a precision of 0.63 and a recall of 0.65 on the development set, and 0.65 precision and 0.67 recall on the test set. The authors also compared several pre-trained models, with the fine-tuned roberta-large model outperforming others like bert-base-uncased, bert-base-multilingual-cased, albert-base, roberta-base, and xlm-roberta-base on the development set.

Limitations and Future Work

The authors acknowledge that their experimentation did not pursue a finer-grained classification within the multi-label task. For future work, they suggest pivoting towards fine-grained multi-label classification, which could involve optimizing the loss function or implementing multi-level classification techniques to enhance the model's generalization capabilities.

Summary updated

BibTeX
@inproceedings{dao2024puer,
  author = {Jiaxu Dao and Zhuoying Li and Youbang Su and Wensheng Gong},
  title = {{Puer at SemEval-2024 Task 4: Fine-tuning Pre-trained Language Models for Meme Persuasion Technique Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure