TL;DR
CARE (Conversational Affective Response Retrieval) is a new two-stage framework designed to help AI systems pick the perfect meme response for an online conversation. By combining a semantic "backbone" (CLIP) with automatically extracted emotion descriptors and training on real-world engagement data (likes), CARE significantly outperforms standard vision-language models in retrieving contextually and affectively appropriate memes.
What Problem Does This Paper Address?
Internet memes are more than just images with text; they are "pragmatic acts" used to express stance, irony, and emotion. Standard retrieval systems often rely on literal image-text similarity—for example, matching a query about "fire" to an image of a literal fire. However, in a conversation, a user might post "This is fine" (referencing the famous dog-in-a-burning-room meme) to express resignation, not an interest in combustion.
The authors identify a major gap: current models struggle with the "affective" (emotional) dimension of memes. A successful meme retrieval system must account for how visual cues and text communicate a specific emotional stance within a conversational thread. As shown in Figure 1, understanding a meme often requires contextual interpretation beyond simple object recognition.
Methods
The researchers propose a two-stage pipeline called CARE:
- Stage 1: CARE-Retriever (The Bi-Encoder): This stage performs efficient coarse retrieval. It uses a CLIP backbone but augments it with "emotion descriptors" (e.g., "sarcastic," "amused") extracted by Large Language Models (LLMs). These descriptors are encoded via BERT and combined with the CLIP image embedding using a gated fusion mechanism. This allows the model to learn how much weight to give to the "vibe" of the meme versus its literal content. It is trained using a multi-positive contrastive loss, recognizing that one conversational prompt can have many valid meme responses.
- Stage 2: CARE-Reranker (The Cross-Encoder): The top candidates from the retriever are passed to a Jina-based cross-encoder. This model looks at the fine-grained interactions between the query text and the candidate image. Crucially, it is trained using LambdaRank with implicit feedback: memes that received more "likes" in actual online threads are ranked higher.
The overall workflow, from raw data to a ranked list of responses, is visualized in Figure 4.
Datasets
The authors introduced a large-scale benchmark consisting of two language-specific collections:
- Douban (Chinese): 13,387 meme records across 1,836 topics, characterized by high-context, culturally implicit humor.
- Imgflip (English): 85,119 meme records across 3,747 topics, featuring high template diversity.
Unlike existing datasets that focus on simple classification (e.g., "is this meme hateful?"), this benchmark preserves the thread structure of online discussions, allowing the model to learn from real query-response pairs. They used LLMs (Doubao Vision Pro for Chinese, Gemini Pro for English) to automatically generate structured affective labels for every meme, as shown in Figure 2.
Key Insights and Ideas
- Affect as a First-Class Citizen: The study proves that "affective consistency"—whether the image and text convey a coherent emotional orientation—is a significant predictor of user engagement.
- The Power of Implicit Feedback: By using "likes" as a supervision signal for the reranker, the model learns to prioritize memes that have a proven track record of "social resonance" rather than just semantic relevance.
- Gated Fusion vs. Simple Concatenation: Instead of just tacking emotion labels onto the end of a vector, the gated mechanism allows the model to dynamically adjust the influence of affect based on the specific image and query.
- Beyond One-to-One Matching: The "multi-positive" training approach acknowledges the open-ended nature of conversation. There isn't one "right" meme; there is a neighborhood of appropriate responses.
Key Results
CARE consistently outperformed strong baselines like VSE++, BLIP, and fine-tuned CLIP across almost all metrics.
- Hit@10 Performance: CARE achieved 24.03% on Douban and 49.33% on Imgflip, compared to 23.16% and 41.07% for fine-tuned CLIP.
- Top-Rank Gains: The improvements were most dramatic at the very top of the results list. On Douban, Hit@1 rose from 10.40% to 13.63% (a 31% relative improvement).
- Qualitative Superiority: Qualitative analysis (Figure 5) shows that while vanilla CLIP might retrieve images based on literal keywords (like "flames"), CARE successfully identifies memes that match the tone of the query, such as mockery or frustration.
Limitations and Future Work
The authors are transparent about several hurdles:
- Noisy Labels: "Likes" are a proxy for relevance but are heavily influenced by external factors like when a meme was posted or how many people saw the thread.
- Source-Image Bias: The current evaluation protocol gives high priority to recovering the original "source" meme of a thread, which might overshadow the model's ability to pick new, creative responses.
- LLM Dependency: The quality of the affective descriptors depends on the underlying LLM used for annotation.
- Future Directions: The team suggests moving toward human-centered validation (asking real people to grade the "funniness" or "appropriateness" of responses) and exploring cross-lingual transfer to see if a model trained on English memes can understand Chinese "vibe" and vice versa.
Summary updated