TL;DR
This paper tackles the challenging task of identifying the roles (hero, villain, victim, other) of entities within internet memes, which often involve complex multimodal interactions and cultural context. They propose a system that leverages pre-trained multimodal encoders and introduces novel sampling strategies to effectively handle the significant class imbalance present in meme datasets, leading to improved performance.
What Problem Does This Paper Address?
Internet memes are a rich source of multimodal communication, combining images and text (often extracted via OCR) to convey humor, commentary, and cultural insights. The task of Semantic Role Labeling (SRL) in this context is particularly tricky because memes often feature:
- Entanglement: The image and text can contradict or subtly influence each other, requiring a deep understanding of both modalities.
- Pragmatics and Context: Interpretation heavily relies on common sense, cultural knowledge, and the specific community where the meme is shared, making it difficult for models to grasp the intended meaning.
- Class Imbalance: In meme datasets, certain roles (like "other") are vastly over-represented compared to others (like "hero" or "victim"), posing a significant challenge for standard machine learning models that tend to favor the majority class.
This paper addresses the CONSTRAINT'22 shared task, which aims to classify entity roles in memes, by proposing solutions that can handle these multimodal complexities and the critical issue of class imbalance.
Methods
The authors build their system upon several pre-trained multimodal encoders, including:
- CLIP (Contrastive Language-Image Pre-training): Known for its strong zero-shot capabilities and ability to align text and image embeddings.
- VisualBERT: A model designed for multimodal tasks, using self-attention to align text and image regions.
- OFA (One For All): A unified sequence-to-sequence framework trained on a diverse set of tasks, aiming for broad applicability.
They then employ three types of classifiers on top of these encoders:
- Multilayer Perceptron (MLP): A standard two-layer MLP used when the encoder outputs a fixed-size representation.
- Attention Mechanism: Used when encoder outputs are not pooled, allowing the entity to query the image and OCR representations.
- Seq2seq: Leveraged with the OFA encoder, framing the task as generating label tokens.
A key contribution is their approach to class imbalance. Instead of simple loss weighting, they explore various sampling strategies:
- Micro: Optimizes for Micro-F1, favoring the majority class.
- Macro: Subsamples to achieve a uniform label distribution, aiming for balanced performance across all classes.
- In-between: A compromise between micro and macro.
- Interpolate & Cycle: Dynamic strategies that adjust sampling rates during training, showing superior performance.
Finally, they utilize ensembling of multiple models to further boost performance.
Datasets
The primary dataset used is the CONSTRAINT'22 dataset. This dataset consists of (meme, entity) pairs, where each meme is represented by an image and its OCR'd caption. The task is to assign one of four roles to each entity: hero, villain, victim, or other. The dataset covers two domains: COVID-19 and US politics. The authors also performed human annotation on a sample of 100 (image, entity) pairs to evaluate the task's difficulty.
Key Insights and Ideas
- Multimodality is Crucial: The paper strongly emphasizes that understanding memes requires more than just text or image analysis; the interplay between them is key. The OCR is often noisy, and entities might not be visually present, necessitating robust multimodal fusion.
- Dynamic Sampling Outperforms Static: The most surprising finding regarding class imbalance is that dynamic sampling strategies (like "interpolate" and "cycle") significantly outperform static ones (like "micro" and "macro"). This suggests that adapting sampling during training is vital for imbalanced datasets.
- CLIP + MLP is a Strong Baseline: The CLIP encoder combined with an MLP classifier emerges as a very strong standalone model, outperforming other encoder-classifier combinations. This highlights CLIP's effectiveness for this specific multimodal task.
- Ensembling is Key for Top Performance: As is common in many NLP and multimodal tasks, ensembling multiple models, especially when combined with gradient boosted trees (XGBoost), yields the best results, significantly surpassing individual models.
- Cultural Context is Hard: Even human annotators show only "moderate" agreement (Cohen's Kappa of 0.47), indicating the inherent difficulty and subjectivity of meme interpretation, often requiring deep cultural and common-sense knowledge.
Key Results
- The best standalone model, using CLIP with an MLP classifier and a "short cycle" sampling strategy, achieved a Macro-F1 score of 47.0.
- Ensembling models, particularly using an XGBoost approach, further improved performance, reaching 47.6 for the ensemble mean and 47.5 for the ensemble median.
- Human annotators achieved a significantly higher average Macro-F1 score of 0.65, underscoring the task's difficulty for machines.
- Dynamic sampling strategies (interpolate, cycle) consistently outperformed static strategies (micro, macro) for the CLIP MLP model, with the "short cycle" strategy achieving 47.0 Macro-F1.
- Qualitative analysis using PCA on CLIP embeddings showed that inanimate objects tend to be classified as "other," while political entities are often labeled as "villain" or "victim," with some surprising exceptions like the "libertarian party" being portrayed as "hero."
Limitations and Future Work
- Dataset Limitations: The authors note that the dataset provides OCR and entity lists, simplifying the task by removing the need for entity detection or OCR correction. In a real-world scenario, a more comprehensive system would handle these aspects. They also suggest that metadata like source website and publication date could aid human and algorithmic annotators.
- Annotation Agreement: The "moderate" inter-annotator agreement (Cohen's Kappa = 0.47) highlights the subjective nature of meme interpretation and the challenge of creating a definitive ground truth.
- Future Work:
- Entity Linking: Resolving surface forms to entity identifiers (e.g., "US" to "United States").
- OCR Normalization: Improving robustness to OCR errors and meme-specific syntax.
- Image Embedding Refinement: Potentially removing text from images to get cleaner visual embeddings.
- Global and Contextualized Entity Embeddings: Training embeddings that are shared across the dataset and also context-specific to the meme's image and OCR.
Summary updated