TL;DR
This paper tackles the challenge of classifying memes into multiple persuasion techniques, organized in a hierarchy, across different languages without prior training data (zero-shot). The team "fralak" developed an ensemble model featuring an RNN that generates hierarchical label paths, achieving surprisingly good generalization to unseen languages despite a modest performance on English.
What Problem Does This Paper Address?
Classifying text into multiple categories simultaneously (multilabel classification) is already complex. When these categories are organized hierarchically (e.g., a broad category like "Emotional Appeal" might have sub-categories like "Fear" or "Sympathy"), it becomes even more challenging. This paper focuses on SemEval-2024 Task 4, specifically Subtask 1, which involves identifying 20 persuasion techniques within meme text. The core difficulties are:
- Hierarchical Structure: The relationships between labels (parent-child) need to be leveraged for better classification.
- Multilabel Nature: A single meme can employ multiple persuasion techniques.
- Zero-Shot Multilingual: The system must perform well on languages (Bulgarian, North Macedonian, Arabic) for which no labeled training data is provided, relying solely on English training data. This is a significant hurdle as language nuances and cultural contexts can drastically affect persuasion techniques.
- Meme Data: Memes often contain informal language, slang, and cultural references, making standard NLP techniques less effective.
Methods
The "fralak" team proposed an ensemble model with three main components:
- Label Restructuring and RNN: The most innovative aspect is transforming the hierarchical labels into strings that represent the full path from the root to each leaf node in the hierarchy. An RNN (Recurrent Neural Network) is then trained to generate these structured label strings. The idea is that the RNN will learn the relationships between labels at different levels of the hierarchy, which is expected to generalize well across languages.
- Basic Neural Network Modules: Two additional modules consist of simple fully connected neural networks. These are designed to complement the RNN and potentially mitigate error propagation issues common in sequence generation.
- Multilingual Sentence Embeddings: To handle the multilingual aspect and informal meme language, the text is represented using multilingual sentence embeddings. This allows the model to capture semantic meaning across different languages.
The system also incorporated preprocessing steps like spelling normalization and Named Entity Recognition (NER), with a specific strategy to replace recognized person entities with a generic token ("Mark") to handle out-of-vocabulary names.
Datasets
- Training/Validation/Development Data: Provided in English for Subtask 1.
- Test Data: Included English, Bulgarian, North Macedonian, and Arabic.
The paper notes that the gold labels for the development set became available late, and the team used the validation set for preliminary testing before combining it with the test set for some experiments.
Key Insights and Ideas
- RNN for Hierarchical Paths: The core idea of using an RNN to generate structured label paths is a novel approach to explicitly encode hierarchical information. This seems to be the key to their cross-lingual generalization.
- Generalization over English Performance: The model performed modestly on the English test set but significantly outperformed many other systems in the zero-shot multilingual setting. This suggests that the learned hierarchical relationships are more robust than language-specific features.
- Preprocessing's Limited Impact on Generalization: Surprisingly, removing NER preprocessing actually improved performance in some languages, suggesting that generic replacements like "Mark" might not always be beneficial and could even hinder generalization if not language-aware.
- RNN's Precision vs. Recall Trade-off: The RNN module exhibited high precision but low recall, meaning it was good at predicting correct labels when it did predict them, but often stopped too early (due to the EOS token or max string length). Increasing the max string length improved precision.
Key Results
- The "fralak" system achieved a hierarchical F1 score of 0.56 on the English test set, ranking 25/33.
- In the zero-shot multilingual setting, the system achieved F1 scores of 0.46 (Bulgarian), 0.46 (North Macedonian), and 0.43 (Arabic).
- Despite lower absolute scores, the system's ranking on the zero-shot tasks was significantly better (10/20 for Bulgarian, 4/20 for North Macedonian, 3/20 for Arabic), indicating superior generalization compared to other teams.
- Experiments showed that the full ensemble model (including the RNN) generally performed better than individual modules, especially for Macedonian and Arabic.
Limitations and Future Work
- English Performance: The model's performance on the English test set was below average.
- Preprocessing Effectiveness: The impact of preprocessing steps like NER on generalization was found to be less significant than initially hypothesized.
- RNN's EOS Token Issue: The RNN's tendency to prematurely generate the End-of-Sequence (EOS) token led to low recall.
- Future Directions:
- Explore different types of embeddings.
- Investigate ways to mitigate the RNN's tendency to generate EOS tokens too easily, potentially by raising a "barrier" or implementing attention mechanisms.
- Develop more language-aware NER preprocessing.
- Experiment with more sophisticated RNN architectures (GRU, LSTM) or combine them with convolutional models.
Summary updated