Internet memes are a dominant form of online communication, yet they present a unique challenge for artificial intelligence. Unlike standard image-text pairs where the text describes the image (such as a caption for a photo), the meaning of a meme is rarely a simple sum of its parts. Instead, memes rely on emergent semantics—a new, synergistic meaning born from the non-linear interplay between a visual trope and a textual phrase.
If you analyze the text in isolation, it might seem completely benign. If you look at the image alone, it may depict a mundane scene from a movie or a cartoon. But when fused, they can express complex humor, deep sarcasm, or insidious hostility. To bridge this semantic gap, researchers have shifted away from unimodal analysis toward multimodal fusion architectures designed to capture the cross-modality friction that gives memes their meaning.
The Limits of Unimodal Analysis
To understand why simple approaches fail, consider how a standard machine learning model processes a meme. If a model uses a text-only classifier, it misses the visual context that provides the punchline or the target of a joke. Conversely, an image-only classifier cannot decode the specific narrative twist introduced by the text.
As a result, early research established that treating these modalities as independent signals creates a hard ceiling on performance. For instance, when predicting meme popularity or generating descriptions, William Yang Wang et al. demonstrated in their foundational work, I Can Has Cheezburger? A Nonparanormal Approach to Combining Textual and Visual Information for Predicting and Generating Popular Meme Descriptions, that visual content, textual content, and social context must be modeled jointly to capture how memes spread and resonate with audiences.
Architectures for Cross-Modality Interplay
The core engineering challenge in meme analysis is fusion: how and when should the visual and textual features be combined?
Multi-Branch Parallel Processing
A foundational strategy relies on multi-branch architectures. These models process text and images through separate, specialized deep learning pipelines before projecting them into a shared semantic space.
[Meme Image] ---> [CNN / ResNet / VGG] ---\
===> [Fusion Layer] ---> [Classification]
[Meme Text] ---> [LSTM / BERT / ALBERT] --/
For example, Ingroj Shrestha et al. proposed a multi-branch model in NLP_UIOWA at SemEval-2020 Task 8: You’re Not the Only One Cursed with Knowledge - Multi Branch Model Memotion Analysis that uses Convolutional Neural Networks (CNNs) for visual feature extraction and Bidirectional Long Short-Term Memory (BLSTM) networks for text. Similarly, Gretel Liz De la Peña Sarracén et al. paired VGG19 with BERT in PRHLT-UPV at SemEval-2020 Task 8: Study of Multimodal Techniques for Memes Analysis to demonstrate that combining these pre-trained representations consistently outperforms single-modality baselines.
Advanced Fusion and Bi-Directional Flow
Simple concatenation of features often fails to capture subtle ironies. To solve this, researchers developed more aggressive fusion techniques. In DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis, Pradyumna Gupta et al. explored bi-fusion techniques. Instead of just merging features at the very end, bi-fusion allows information to flow bidirectionally between the text and image representations during the learning process, forcing the model to alter its interpretation of the text based on the image, and vice versa.
Decoding Metaphor via Multi-Task Learning
Because memes function as visuo-lingual metaphors, classifying them requires models to understand abstract concepts like sarcasm, offense, and humor simultaneously. Yingmei Guo et al. focused explicitly on this in Guoym at SemEval-2020 Task 8: Ensemble-based Classification of Visuo-Lingual Metaphor in Memes, using ensemble methods to capture how visual imagery maps onto linguistic meanings.
To make these models robust, researchers frequently turn to multi-task learning. The intuition here is that a model cannot truly understand if a meme is "harmful" unless it also understands if it is intended to be "humorous" or "sarcastic."
- In LT3 at SemEval-2020 Task 8: Multi-Modal Multi-Task Learning for Memotion Analysis, Pranaydeep Singh et al. demonstrated that training a system to solve multiple emotional classification tasks at once improves its performance on individual tasks.
- This sentiment is echoed by Irina Bejan in MemoSYS at SemEval-2020 Task 8: Multimodal Emotion Analysis in Memes, where transfer learning and multimodal fusion were leveraged to map the complex emotional landscape of internet humor.
Detecting Harmful Content and Low-Resource Challenges
The most critical application of this research lies in content moderation—specifically, identifying misogyny, trolling, and disinformation. Harmful memes are frequently designed to bypass automated text filters by hiding their offensive nature behind an innocent image, or using an innocent phrase over a derogatory image.
Co-Representation and Specialized Modules
To unmask these tactics, Yimeng Gu et al. introduced a multi-modal, multi-task Variational Autoencoder (MMVAE) in MMVAE at SemEval-2022 Task 5: A Multi-modal Multi-task VAE on Misogynous Meme Detection. The MMVAE forces the model to learn a joint distribution of both text and images, making it highly sensitive to mismatches or toxic combinations. Other approaches, like the one proposed by Pablo Cordon et al. in I2C at SemEval-2022 Task 5: Identification of misogyny in internet memes, complement text embeddings from BERT with specialized visual modules (such as nudity or object detection) to flag problematic visual cues.
The Challenge of Regional Languages
Meme analysis becomes significantly harder when moving away from English to regional or low-resource languages, where cultural context changes rapidly. Bharathi Raja Chakravarthi et al. organized a shared task detailed in Overview of Shared Task on Multitask Meme Classification - Unraveling Misogynistic and Trolls in Online Memes to target misogynistic content in Tamil and Malayalam memes.
To tackle these specific linguistic and visual combinations:
- Bo Huang et al. developed a BiGRU and CNN fusion architecture in HUB@DravidianLangTech-EACL2021: Meme Classification for Tamil Text-Image Fusion.
- Sidharth Mahesh et al. utilized language-specific transformers like MuRIL alongside ResNet-50 in MUCS@LT-EDI-2024: Exploring Joint Representation for Memes Classification to capture regional nuances.
- For broader societal tracking, such as evaluating public sentiment on national defense, Nirmalendu Prakash et al. introduced the TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore dataset, proving that multimodal fusion is vital for understanding localized political discourse.
Beyond Classification: Persuasion and Topic Modeling
As the field matures, the focus is expanding from basic emotion classification to understanding intent and structure. In CVcoders on Semeval-2024 Task 4, Fatemezahra Bakhshande et al. pushed the boundaries of multimodal analysis by attempting to detect fine-grained persuasion techniques within multilingual memes, showing how visual framing and rhetorical text work hand-in-hand to manipulate public opinion.
Ultimately, the techniques perfected in meme analysis are influencing how we understand the broader digital landscape. Memes are simply a hyper-dense manifestation of the modern web, where text and images are inherently connected. By mastering the art of text-image fusion for memes, researchers are unlocking the ability to parse the complex, multi-layered communication formats that define human interaction online.
Key Foundational Work
NLP_UIOWA at SemEval-2020 Task 8: You’re Not the Only One Cursed with Knowledge - Multi Branch Model Memotion Analysis (2020) — contributed 2 concept elements to this direction.
Memebusters at SemEval-2020 Task 8: Feature Fusion Model for Sentiment Analysis on Memes Using Transfer Learning (2020) — contributed 1 concept elements to this direction.
PRHLT-UPV at SemEval-2020 Task 8: Study of Multimodal Techniques for Memes Analysis (2020) — contributed 1 concept elements to this direction.