TL;DR
This paper tackles the challenge of automatically identifying persuasive techniques in memes, a common vehicle for misinformation. The UMUTeam developed a multimodal approach using large language models (LLMs) and a vision-language model (LlaVa) to analyze both text and images, achieving competitive results in a SemEval task. Their findings highlight the effectiveness of incorporating image descriptions generated by LLMs for improved multimodal performance.
What Problem Does This Paper Address?
The proliferation of social media has unfortunately led to a surge in misinformation and disinformation. Memes, with their potent combination of images and text, have become a particularly effective tool for spreading such content. These visual-textual narratives often employ subtle rhetorical and psychological techniques to influence public perception and beliefs, making them difficult to analyze and combat. The SemEval-2024 Task 4 aimed to address this by developing models capable of identifying persuasion techniques within memes, considering both unimodal (text-only) and multimodal (text and image) settings. This task is crucial for understanding and mitigating the impact of propaganda and manipulation online.
Methods
The UMUTeam's approach centers on fine-tuning pre-trained Transformer models, specifically ROBERTa-large, for classification tasks. For the unimodal subtask (Subtask 1), only the textual content of the memes was used.
For the multimodal subtasks (Subtask 2a and 2b), they employed a Large Multimodal Model (LMM) called LlaVa (Liu et al., 2023). LlaVa was used to generate textual descriptions of the meme images. These generated descriptions were then concatenated with the original meme text. This combined textual information was then fed into the fine-tuned RoBERTa-large model for classification. This strategy of using LLM-generated image descriptions, rather than directly merging image and text embeddings, proved to be an effective way to leverage multimodal information.
The task involved three subtasks:
- Subtask 1: Unimodal hierarchical multi-label classification of 20 persuasion techniques using text only.
- Subtask 2a: Multimodal hierarchical multi-label classification of 22 persuasion techniques using text and image.
- Subtask 2b: Multimodal binary classification to determine if a meme contains any persuasion technique.
Datasets
The researchers utilized the dataset provided by the organizers for SemEval-2024 Task 4. This dataset includes texts and images labeled with corresponding persuasion techniques. For Subtask 2b, it also includes a binary annotation indicating the presence of persuasion. The dataset was divided into training, development, and validation sets. The UMUTeam combined the training and development sets for their training phase. The training dataset contained 8,000 examples for Subtasks 1 and 2a, and 1,499 examples for Subtask 2b.
Key Insights and Ideas
- LLM-generated image descriptions enhance multimodal performance: The core insight is that using LlaVa to generate textual descriptions of meme images and then combining these with the meme's text significantly improved performance in multimodal settings compared to other fusion strategies. This suggests that translating visual information into a textual format that can be processed alongside existing text is a powerful approach.
- Concatenation over complex fusion: The paper found that simply concatenating the text and the LLM-generated description yielded better results and required fewer resources than more complex methods of merging image and text embeddings. This points to the efficiency and effectiveness of this straightforward approach.
- Hierarchical classification is complex: The error analysis revealed that the model struggled with certain persuasion techniques, particularly those with ambiguous or overlapping features, suggesting that the hierarchical nature of the classification and potential class imbalance pose significant challenges.
Key Results
The UMUTeam achieved competitive results across all three subtasks:
- Subtask 1 (Unimodal Text): Ranked tenth with a Hierarchical F1 (H-F1) score of 64.774%.
- Subtask 2a (Multimodal Hierarchical): Ranked fourth with an H-F1 score of 69.003%.
- Subtask 2b (Multimodal Binary): Ranked eighth with a Macro F1 (M-F1) score of 78.660%.
Their multimodal approach in Subtask 2a outperformed the baseline by 24.297% and was only 5.589% away from the top-performing team. Similarly, in Subtask 2b, their system improved by up to 53.66% compared to the baseline and was only 2.37% behind the first place.
Limitations and Future Work
The authors acknowledge that the organizers did not provide gold labels for the test set, so their error analysis was based on the development set. They identified specific classes where their model struggled, such as "Misrepresentation of Someone's Position (Straw Man)" and "Obfuscation, Intentional vagueness, Confusion," which they attribute to potential class imbalance or insufficient representative features.
For future work, the team proposes to:
- Evaluate the relationship between persuasion techniques across different domains.
- Re-annotate existing Spanish datasets (SatiCorpus 2021 and PoliticES 2022) with the 22 persuasion techniques to evaluate binary and hierarchical multi-label classification approaches on these new datasets.
- Explore the identification of misogyny using similar techniques, as persuasion techniques are often present in such content.
Summary updated