Abstract
This paper introduces a novel framework designed to capture "multimodal contrast"—the emergent, non-compositional meaning that arises when images and text conflict, such as in irony, humor, or hateful memes. By combining a Cross-modal Memory module with a Reinforced Contrast Recognition (RCR) strategy using self-rejection sampling, the authors enable Large Language Models (LLMs) to interpret complex social media content where the whole is significantly different from the sum of its parts. The proposed model, MemRCR, achieves state-of-the-art results across multiple benchmarks, including the Hateful Meme Challenge.
The Challenge of Multimodal Contrast
Most multimodal AI research focuses on image-text matching, ensuring models recognize that a word like "dog" corresponds to a visual representation of a dog. However, real-world social media content often utilizes purposeful misalignment. In a hateful meme, the image and text might be innocuous in isolation; the harmful intent only emerges from the contrast between the two modalities.
Current models often struggle with this "non-compositional" effect because they are trained primarily to find similarities. The authors argue that memes function similarly to idioms: just as "crossing the Rubicon" means more than walking across a river, a meme's meaning is an emergent property that requires explicit modeling of conflicting relationships to uncover "extended semantics."
Proposed Architecture: MemRCR
The authors propose an encoding-decoding approach built on top of an LLM backbone (such as Vicuna or MiniGPT-4). The architecture introduces two primary innovations to handle contrastive semantics:
1. Cross-modal Memory (M)
This module utilizes a matrix of trainable "memory vectors." When an image-text pair is processed, the model calculates similarity scores between the input and these vectors. It then ranks and samples the most relevant vectors to create a contrast vector ($m_x$). This vector acts as a soft prompt, providing the LLM with specialized context regarding the interaction between the modalities.
2. Reinforced Contrast Recognition (RCR)
Because there is no "gold standard" for a perfect contrast vector, the authors employ Reinforcement Learning. They train a Reward Model to distinguish between "contrasted" pairs (actual memes) and "aligned" pairs (images paired with literal captions generated by off-the-shelf tools). During training, the system uses self-rejection sampling to generate multiple candidate memory vectors. The reward model selects the candidate showing the strongest contrast, which is then used to optimize the memory module.
Experimental Results
The approach was evaluated on four benchmark datasets: HMC (Hateful Meme Challenge), Memotion7k (sentiment/humor), MultiOFF (offensive content), and Memeplate (Chinese humor recognition).
Key findings include:
- State-of-the-Art Performance: MemRCR achieved 77.88% Accuracy and 86.34 AUROC on the HMC dataset, outperforming strong baselines like CLIP and Flamingo.
- Robustness: The RCR module consistently improved performance across different LLM backbones, proving the strategy's effectiveness regardless of the underlying model size.
- Qualitative Success: Case studies demonstrated that MemRCR could correctly identify negative sentiment in ironic memes (e.g., positive text paired with a riot image) where standard models were misled by the literal text.
Conclusion and Future Work
MemRCR demonstrates that "contrast" is a distinct semantic feature that can be modeled through specialized memory and reinforcement strategies. While the model excels at social media analysis, future work is required to address the computational overhead of self-rejection sampling and to adapt the definition of "contrast" for other domains, such as medical imaging or technical documentation.
Summary updated