NLPNCHU at SemEval-2024 Task 4: A Comparison of MDHC Strategy and In-domain Pre-training for Multilingual Detection of Persuasion Techniques in Memes

Shih-wei Guo, Yao-chung FanNAACL 2024

TL;DR

This paper tackles the challenge of automatically identifying persuasive techniques in multilingual memes, a complex multimodal task. The authors propose a hierarchical multi-label classification (HMC) approach, leveraging domain-specific pre-training and multimodal fusion (text + image), achieving competitive results in the SemEval-2024 task.

What Problem Does This Paper Address?

Persuasive discourse, aiming to influence beliefs and behaviors, is prevalent in advertising and propaganda. Memes, as a popular form of online communication, often employ sophisticated persuasive techniques, making them potent tools for spreading ideas and misinformation. Identifying these techniques is crucial for understanding online influence and combating manipulation. However, this task is challenging due to:

  • Multimodality: Memes combine visual and textual elements, both of which contribute to persuasion.
  • Multilingualism: Persuasive techniques can manifest differently across languages.
  • Hierarchical Nature: Persuasive techniques are not isolated but organized in a hierarchical structure, where broader categories encompass more specific ones.
  • Data Imbalance: Datasets for such tasks often suffer from skewed distributions of positive and negative examples, as well as imbalances across different technique categories.

The SemEval-2024 Task 4 specifically addresses this by providing a dataset and a hierarchical structure (HDAG) for detecting 22 persuasion techniques in multilingual memes.

Methods

The core of the proposed solution is a Multi-Dimensional Hierarchical Classification (MDHC) strategy. This approach transforms the problem into an HMC framework, allowing for the simultaneous handling of textual and visual data. Key methodological components include:

  • Hierarchical Multi-label Classification (HMC): The 22 persuasion techniques are structured into a three-level hierarchy: Root (presence of any technique), First Layer (Ethos, Logos, Pathos), Second Layer (Ad Hominem, Justification, Distraction, Simplification, Other), and Leaf Nodes (the 22 specific techniques).
  • Multimodal Fusion: To capture both text and image information, the authors employ the CLIP model. CLIP's independent text and image encoders are utilized, with the text encoder being a XLM-ROBERTA-large-twitter model fine-tuned on domain-specific meme data. This allows for simultaneous processing of visual and textual features.
  • Domain-Specific Pre-training: The XLM-ROBERTA-large-twitter model was pre-trained on a large corpus of multilingual meme text, aiming to improve its understanding of meme-specific language and persuasive nuances.
  • MDHC Strategies: Two main MDHC strategies were compared:
  • Stacking + GC (Global Classifier): A global classifier is applied at the first phase, and a stacking strategy combines predictions in the second phase.
  • Stacking + LCL (Local Classifier per Level): Local classifiers are trained for each level of the hierarchy in the first phase, with a stacking strategy used in the second phase.
  • Internal Negative Data Augmentation (INDA): To address data imbalance, particularly the scarcity of non-persuasive examples, INDA was employed. This involves creating negative examples based on the hierarchical structure, ensuring labeling consistency.
  • Distribution-Balanced Loss (DBL): To mitigate issues arising from label co-occurrence and the dominance of negative labels in long-tail distributions, the DBL loss function was used.

Datasets

The primary dataset used is from the SemEval-2024 Task 4, containing approximately 15,000 memes in English and other languages. This dataset is used for training and evaluation across different subtasks. The authors also reference the PTC2 dataset for potential augmentation, though they note its limitations for hierarchical multi-label classification. For Subtask 2b, a balanced dataset of hate memes collected by Meta AI (Harmful Memes Dataset) was utilized for binary classification.

Key Insights and Ideas

  • Domain-Specific Pre-training is Crucial: Fine-tuning XLM-ROBERTA on meme-specific text significantly boosted performance, highlighting the importance of adapting language models to the target domain.
  • Multimodal Fusion is Essential: Combining text and image embeddings via CLIP is vital for accurately detecting persuasive techniques, as visual elements often reinforce or convey persuasive tactics.
  • Hierarchical Classification is Beneficial: Structuring the problem within an HMC framework effectively leverages the inherent relationships between persuasion techniques, leading to better performance.
  • Addressing Data Imbalance is Key: Techniques like INDA and DBL loss are critical for handling the skewed distributions common in persuasion detection datasets, preventing models from being biased towards majority classes.
  • Stacking + GC Outperforms: The Stacking + GC strategy generally demonstrated superior performance compared to Stacking + LCL, suggesting that a global approach to classification at the initial stage is more effective for this task.

Key Results

  • Subtask 1 (Text-level persuasion detection): The proposed method, particularly with domain-specific pre-training and the Stack + GC strategy, achieved competitive rankings. For Arabic, the system secured 2nd place in Subtask 1. On average across four languages (English, Bulgarian, North Macedonian, Arabic), the method ranked 4th in the test set.
  • Subtask 2a (Multimodal persuasion detection): The multimodal approach using CLIP and the fine-tuned XLM-ROBERTA-Twitter model, combined with the Stack + GC strategy, showed strong performance, achieving competitive scores across languages.
  • Subtask 2b (Binary classification of persuasive techniques): The multimodal model achieved competitive scores, with the fine-tuned text encoder yielding better results.

Limitations and Future Work

  • Cross-Lingual Transfer Limitations: The study notes that while the multilingual model helps, direct cross-lingual transfer for persuasion technique identification is not perfect. The model's performance in non-English languages is influenced by the availability of training data and the inherent complexities of cross-lingual representation.
  • Cultural Divergence: Differences in cultural contexts across languages can lead to discrepancies in how memes are perceived and interpreted, posing a challenge for multilingual persuasion detection.
  • Future Work: The authors suggest further exploration of multimodal models and strategies to improve cross-lingual capabilities. Investigating more advanced techniques for handling long-tail distributions and label co-occurrence could also yield further improvements. The effectiveness of domain-specific pre-training is a strong indicator for future research directions.

Summary updated

BibTeX
@inproceedings{guo2024nlpnchu,
  author = {Shih-wei Guo and Yao-chung Fan},
  title = {{NLPNCHU at SemEval-2024 Task 4: A Comparison of MDHC Strategy and In-domain Pre-training for Multilingual Detection of Persuasion Techniques in Memes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure