Abstract
ArchiMeDe is a multimodal ensemble architecture designed to distinguish internet memes from standard images by fusing Italian language models with multiple computer vision perspectives. By combining text, image embeddings, and social metadata, the system demonstrates how transfer learning can achieve high performance in nuanced social media tasks without requiring massive computational resources.
Introduction
Detecting memes is a deceptively complex task for artificial intelligence. Unlike standard object recognition, a meme’s meaning emerges from the intersection of visual templates, overlaid text, and deep cultural context—often involving irony or satire. This paper presents ArchiMeDe (Architecture for Meme Detection), a system developed for the DANKMEMES task of the EVALITA 2020 campaign. The challenge focuses on the Italian political scene, requiring models to differentiate between standard photographs of politicians and memes featuring those same individuals. This necessitates a system that can simultaneously "see" the image, "read" the text, and "understand" the social context.
Methodology
The ArchiMeDe system utilizes a "late-fusion" ensemble approach to process multimodal inputs. The architecture is divided into three primary components:
- Textual Branch: Uses UmBERTo, a RoBERTa-based language model pre-trained on Italian text, to generate a single vector representing the meme's caption.
- Visual Branch: To capture diverse visual features, the system employs an ensemble of three distinct architectures: ResNet-50, AlexNet, and DenseNet.
- Metadata Integration: The model incorporates real-world signals, including standardized user engagement metrics, normalized post dates, and one-hot encoded lists of visual actors (politicians) present in the frame.
For each of the three vision models, a multimodal embedding is created by concatenating the text, image, and metadata vectors. This combined vector is processed through an 8-layer feed-forward neural network. The final classification (Meme vs. Not-Meme) is determined by a majority vote among the three branches.
Results and Discussion
ArchiMeDe achieved an F1 score of 0.7664 in the competition, ranking 7th overall. A key insight from the development process was the "Multi-View" advantage: different vision models made different errors, and the ensemble approach compensated for individual architectural weaknesses. Quantitative analysis showed that the ArchiMeDe ensemble consistently outperformed single-encoder variants.
Qualitatively, the model was highly effective at identifying images with manual edits and speech bubbles. However, standard computer vision data augmentation (e.g., rotation or zooming) was found to degrade performance, likely because these transformations can obscure the specific "meme-ness" of a template or render overlaid text unreadable. The primary remaining challenges involve deep cultural irony and sophisticated "face-swapping" techniques, which require higher levels of semantic understanding.
Conclusion
The ArchiMeDe architecture demonstrates the efficacy of ensembling pre-trained models for complex multimodal classification. Future work will explore replacing the simple majority vote with a trained aggregator network (meta-learner) and expanding the system's linguistic capabilities beyond Italian.
Summary updated