Abstract
This paper presents a robust solution to the Memotion 3.0 Shared Task. The goal of this task is to classify the emotion and the corresponding intensity expressed by memes, which are usually in the form of images with short captions on social media. Understanding the multi-modal features of the given memes will be the key to solving the task. In this work, we use CLIP to extract aligned image-text features and propose a novel meme sentiment analysis framework, consisting of a Cooperative Teaching Model (CTM) for Task A and a Cascaded Emotion Classifier (CEC) for Tasks B&C. CTM is based on the idea of knowledge distillation, and can better predict the sentiment of a given meme in Task A; CEC can leverage the emotion intensity suggestion from the prediction of Task C to classify the emotion more precisely in Task B. Experiments show that we achieved the 2nd place ranking for both Task A and Task B and the 4th place ranking for Task C, with weighted F1-scores of 0.342, 0.784, and 0.535 respectively. The results show the robustness and effectiveness of our framework.
Introduction
Memes represent a unique and complex communication challenge for artificial intelligence. Unlike standard image-text pairs, a meme’s meaning is rarely literal; it is inherently multi-modal, often relying on implicit humor, sarcasm, or deep cultural context. The Memotion 3.0 Shared Task addresses this by defining three sub-tasks: Sentiment Classification (Task A), Emotion Identification (Task B), and Intensity Quantification (Task C).
The NYCU-TWO team approached these challenges by treating meme analysis as a hierarchical learning problem. Their framework addresses two primary difficulties: the inherent ambiguity of "neutral" sentiment labels and the logical dependency between an emotion's intensity and its presence.
Methodology
The Meme Encoder Foundation
To capture visual and textual nuances, the framework utilizes a dual-encoder strategy. Features are extracted from a fine-tuned Swin Transformer and a CLIP (Contrastive Language-Image Pre-training) model. CLIP is essential for this task as it aligns visual and textual data into a shared high-dimensional space, allowing the model to better understand the non-literal relationships between a caption and its image.
Cooperative Teaching Model (CTM) for Sentiment
The authors argue that "neutral" sentiment often masks subtle positive or negative leanings. To address this, they implemented a teacher-student framework for Task A.
In the CTM, two "teachers" are trained: a "good" teacher focused on positive sentiment and a "bad" teacher focused on negative sentiment. These teachers use "pre-labels" that merge neutral samples with their respective polarities. Student models then learn to mimic these teachers. To ensure robustness, Gaussian noise is injected into meme embeddings during training; if a student's prediction remains stable despite this noise, the model is considered confident in its classification.
Cascaded Emotion Classifier (CEC) for Emotion and Intensity
For Tasks B and C, the team developed the Cascaded Emotion Classifier (CEC). Rather than treating emotion type and intensity as independent variables, the CEC leverages their natural hierarchy.
The model first predicts the intensity of an emotion (Task C). This prediction then serves as a "suggestion" for the final classifier, helping it decide if a specific emotion—such as humor, sarcasm, or offense—is present at all (Task B).
Experimental Results
The NYCU-TWO framework proved highly effective in the Memotion 3.0 competition, securing top rankings across all sub-tasks:
- Task A (Sentiment): 2nd Place (Weighted F1: 0.342)
- Task B (Emotion): 2nd Place (Weighted F1: 0.784)
- Task C (Intensity): 4th Place (Weighted F1: 0.535)
Ablation studies highlighted the criticality of the CTM; without the teacher-student cooperation and learned thresholds, performance on sentiment analysis dropped by over 26%.
Discussion and Future Work
While the framework demonstrated robustness, several challenges remain. The Memotion 3.0 dataset features "Hinglish" (code-mixed Hindi and English), whereas the foundation models like CLIP were primarily trained on English. Future improvements could involve pre-training on more diverse, code-mixed datasets.
Additionally, the authors observed that CLIP often aligns images with literal descriptions. Since memes are frequently ironic, developing a new contrastive learning objective specifically for non-literal image-text relationships could further enhance performance. Finally, while the CEC architecture benefited Task B, the team noted that task-specific networks occasionally outperformed the combined model, suggesting a need for more refined multi-task optimization.
Summary updated