CARE: Conversational Affective Response Retrieval for Internet Memes

Jiajun Li, Leihan Zhang, Teng Zhang, Qiang YanarXiv 2026 · arXiv

TL;DR

CARE (Conversational Affective Response Retrieval) is a new two-stage framework designed to help AI systems pick the perfect meme response for an online conversation. By combining a semantic "backbone" (CLIP) with automatically extracted emotion descriptors and training on real-world engagement data (likes), CARE significantly outperforms standard vision-language models in retrieving contextually and affectively appropriate memes.

What Problem Does This Paper Address?

Internet memes are more than just images with text; they are "pragmatic acts" used to express stance, irony, and emotion. Standard retrieval systems often rely on literal image-text similarity—for example, matching a query about "fire" to an image of a literal fire. However, in a conversation, a user might post "This is fine" (referencing the famous dog-in-a-burning-room meme) to express resignation, not an interest in combustion.

The authors identify a major gap: current models struggle with the "affective" (emotional) dimension of memes. A successful meme retrieval system must account for how visual cues and text communicate a specific emotional stance within a conversational thread. As shown in Figure 1, understanding a meme often requires contextual interpretation beyond simple object recognition.

Figure 1: Illustrative textual situations and associated memes.

Methods

The researchers propose a two-stage pipeline called CARE:

  1. Stage 1: CARE-Retriever (The Bi-Encoder): This stage performs efficient coarse retrieval. It uses a CLIP backbone but augments it with "emotion descriptors" (e.g., "sarcastic," "amused") extracted by Large Language Models (LLMs). These descriptors are encoded via BERT and combined with the CLIP image embedding using a gated fusion mechanism. This allows the model to learn how much weight to give to the "vibe" of the meme versus its literal content. It is trained using a multi-positive contrastive loss, recognizing that one conversational prompt can have many valid meme responses.
  2. Stage 2: CARE-Reranker (The Cross-Encoder): The top candidates from the retriever are passed to a Jina-based cross-encoder. This model looks at the fine-grained interactions between the query text and the candidate image. Crucially, it is trained using LambdaRank with implicit feedback: memes that received more "likes" in actual online threads are ranked higher.

The overall workflow, from raw data to a ranked list of responses, is visualized in Figure 4.

Overview of the CARE framework

Datasets

The authors introduced a large-scale benchmark consisting of two language-specific collections:

  • Douban (Chinese): 13,387 meme records across 1,836 topics, characterized by high-context, culturally implicit humor.
  • Imgflip (English): 85,119 meme records across 3,747 topics, featuring high template diversity.

Unlike existing datasets that focus on simple classification (e.g., "is this meme hateful?"), this benchmark preserves the thread structure of online discussions, allowing the model to learn from real query-response pairs. They used LLMs (Doubao Vision Pro for Chinese, Gemini Pro for English) to automatically generate structured affective labels for every meme, as shown in Figure 2.

Automatic affective representation pipeline

Key Insights and Ideas

  • Affect as a First-Class Citizen: The study proves that "affective consistency"—whether the image and text convey a coherent emotional orientation—is a significant predictor of user engagement.
  • The Power of Implicit Feedback: By using "likes" as a supervision signal for the reranker, the model learns to prioritize memes that have a proven track record of "social resonance" rather than just semantic relevance.
  • Gated Fusion vs. Simple Concatenation: Instead of just tacking emotion labels onto the end of a vector, the gated mechanism allows the model to dynamically adjust the influence of affect based on the specific image and query.
  • Beyond One-to-One Matching: The "multi-positive" training approach acknowledges the open-ended nature of conversation. There isn't one "right" meme; there is a neighborhood of appropriate responses.

Key Results

CARE consistently outperformed strong baselines like VSE++, BLIP, and fine-tuned CLIP across almost all metrics.

  • Hit@10 Performance: CARE achieved 24.03% on Douban and 49.33% on Imgflip, compared to 23.16% and 41.07% for fine-tuned CLIP.
  • Top-Rank Gains: The improvements were most dramatic at the very top of the results list. On Douban, Hit@1 rose from 10.40% to 13.63% (a 31% relative improvement).
  • Qualitative Superiority: Qualitative analysis (Figure 5) shows that while vanilla CLIP might retrieve images based on literal keywords (like "flames"), CARE successfully identifies memes that match the tone of the query, such as mockery or frustration.
Qualitative comparison of retrieval results

Limitations and Future Work

The authors are transparent about several hurdles:

  • Noisy Labels: "Likes" are a proxy for relevance but are heavily influenced by external factors like when a meme was posted or how many people saw the thread.
  • Source-Image Bias: The current evaluation protocol gives high priority to recovering the original "source" meme of a thread, which might overshadow the model's ability to pick new, creative responses.
  • LLM Dependency: The quality of the affective descriptors depends on the underlying LLM used for annotation.
  • Future Directions: The team suggests moving toward human-centered validation (asking real people to grade the "funniness" or "appropriateness" of responses) and exploring cross-lingual transfer to see if a model trained on English memes can understand Chinese "vibe" and vice versa.

Summary updated

Abstract

Conversational meme retrieval supports response selection in online communication, where relevance depends on both contextual meaning and affective expression. We study this task using two language-specific collections from Douban and Imgflip, containing 98,506 meme records across 5,583 discussion topics. Thread membership supplies weak relevance labels, while likes provide implicit ranking feedback. Automatically extracted sentiment, facial-expression indicators, and cross-modal consistency features support an exploratory analysis of engagement. We propose CARE, a Conversational Affective Response Retrieval framework that combines an emotion-enhanced CLIP retriever with a Jina-based reranker. The retriever incorporates free-form emotion descriptors through gated fusion and learns from multiple positive examples per query. The reranker combines query--image interactions with emotion representations and is trained using engagement-based ranking supervision. Under the reported topic-held-out protocol, CARE achieves Hit@10 of 24.03\% on Douban and 49.33\% on Imgflip, compared with 23.16\% and 41.07\%, respectively, for fine-tuned CLIP. These results support the utility of the complete two-stage system for recovering thread-associated memes. They do not independently establish perceived social resonance or isolate the contribution of affect from reranker training. The study provides a basis for further evaluation of affect-aware retrieval with human relevance judgments and controlled comparisons.
BibTeX
@misc{li2026care,
  author = {Jiajun Li and Leihan Zhang and Teng Zhang and Le Zhang and Qiang Yan},
  title = {{CARE: Conversational Affective Response Retrieval for Internet Memes}},
  year = {2026},
  eprint = {2611.23005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23005},
}
Figure