HierarchyEverywhere at SemEval-2024 Task 4: Detection of Persuasion Techniques in Memes Using Hierarchical Text Classifier

Omid Ghahroodi, Ehsaneddin AsgariNAACL 2024

TL;DR

This paper tackles the challenging task of detecting propaganda techniques in memes, a prevalent form of online communication. By leveraging hierarchical text classification (HTC) and a state-of-the-art model, they achieved top performance in English sub-task 2a, demonstrating the power of structured classification for this complex problem.

What Problem Does This Paper Address?

Propaganda, the deliberate dissemination of biased information, is a growing concern, especially with the rise of social media. Memes, with their combination of text and images, are particularly effective vehicles for spreading propaganda. SemEval-2024 Task 4 aimed to address this by identifying propaganda techniques within memes. This task presented a unique challenge due to the hierarchical nature of propaganda techniques themselves, meaning some techniques are sub-categories of others. Existing text classification methods often struggle with such hierarchical structures, and multimodal approaches (combining text and image) can be complex and not always superior. The authors sought to improve propaganda detection by specifically addressing the hierarchical relationships between techniques.

Methods

The core of the authors' approach is Hierarchical Text Classification (HTC). Instead of treating each propaganda technique as an independent label, HTC models leverage the inherent tree-like structure of these techniques. They employed a state-of-the-art HTC model called HPT (Hierarchy-aware Prompt Tuning), which uses a BERT-based architecture.

To adapt the propaganda technique hierarchy to the HPT model, they transformed the provided Directed Acyclic Graph (DAG) into a hierarchical tree. This involved splitting nodes with multiple parents into new, more specific nodes (e.g., "Whataboutism" with parents "Distraction" and "Ad hominem" became "Distraction_Whataboutism" and "Ad hominem_Whataboutism"). They found that introducing a top-level "propagandistic" vs. "non-propagandistic" split improved performance.

Interestingly, for the multimodal sub-task (Task 2a), they focused solely on the textual content of the memes, disregarding the images. They also incorporated additional datasets for pre-training, including data from Da San Martino et al. (2020) and Dimitrov et al. (2021), noting that the latter improved performance. They also experimented with using a [CLS] token to better capture coherence between multiple sentences within a meme.

Datasets

The primary dataset used was from SemEval-2024 Task 4, which includes memes with associated propaganda technique tags. This dataset was provided in English, Bulgarian, North Macedonian, and Arabic, with English data available in a supervised setting and others in a zero-shot setting.

They also utilized two additional datasets:

  • (Da San Martino et al., 2020): Focused on propaganda detection in news articles. They adapted this by assigning techniques to entire articles if a span contained them, but found this approach led to decreased accuracy.
  • (Dimitrov et al., 2021): Focused on propaganda detection in texts and images, without considering hierarchy. Incorporating this data improved their model's accuracy.

Key Insights and Ideas

  • Hierarchy Matters: Explicitly modeling the hierarchical relationships between propaganda techniques significantly boosts classification accuracy, outperforming simpler approaches.
  • Text-Only Can Win: For multimodal tasks involving memes, focusing solely on the textual component can be more effective than incorporating images, especially when dealing with complex semantic relationships. This suggests that current multimodal models may struggle to fully leverage image information for this specific task.
  • Pre-training is Crucial: Leveraging external datasets, particularly those that align with the task's focus (like Dimitrov et al., 2021), can significantly improve performance.
  • Data Preprocessing Nuances: Using a [CLS] token to bridge sentences within memes improved performance, highlighting the importance of how text is segmented and represented.

Key Results

The authors achieved first place in English Sub-task 2a (multimodal, but using only text), outperforming all other participants. Table 1 shows their system achieved an HF1 score of 0.74592 for English Subtask 2a, with a rank of 1/14. While they performed well in English, their system showed poorer performance in non-English sub-tasks, which they attribute to potential translation errors and the lack of language-specific pre-processing pipelines.

Limitations and Future Work

  • Non-English Performance: The system struggled with languages other than English, indicating a need for better multilingual handling, potentially through more robust translation or language-specific models.
  • Image Neglect: While their text-only approach was successful for Task 2a, it means they didn't fully explore the multimodal aspect. Future work could focus on developing multimodal models that can better integrate image and text information for propaganda detection.
  • Translation Errors: The authors acknowledge that translation errors likely impacted performance in non-English tasks.
  • Future Objective: The authors state their next objective is to "develop a better hierarchical text classification model with better performance," suggesting continued research into improving HTC architectures and training strategies.

Summary updated

BibTeX
@inproceedings{ghahroodi2024hierarchyeverywhere,
  author = {Omid Ghahroodi and Ehsaneddin Asgari},
  title = {{HierarchyEverywhere at SemEval-2024 Task 4: Detection of Persuasion Techniques in Memes Using Hierarchical Text Classifier}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure