SEAF: Selective Semantic Enrichment and Visual Evidence Fusion for Multimodal Sarcasm Detection

Faxian Wan, Xiaocui Yang, Yifan Cao, Shi Feng, Daling Wang, Yifei ZhangarXiv 2026 · arXiv

TL;DR

SEAF (Selective Semantic Enrichment and Visual Evidence Fusion) is a novel framework that achieves state-of-the-art results in multimodal sarcasm detection by intelligently separating "what we see" from "what we need to know." By using Large Language Models (MLLMs) to selectively provide missing entity context while using visual descriptions only for alignment statistics, SEAF avoids the redundancy that plagues simpler "prompt-stuffing" approaches.

What Problem Does This Paper Address?

Sarcasm is the art of saying one thing while meaning another, often relying on a sharp contrast between text and image. Detecting it computationally is notoriously difficult because it frequently requires external knowledge or reading text embedded within an image (like a meme).

While Multimodal Large Language Models (MLLMs) can generate helpful auxiliary information, simply dumping all MLLM-generated text into a model often leads to "information overload." For example, an MLLM might describe a person's suit in detail—information already captured by visual features—while failing to explain the specific political scandal that makes the post sarcastic. Furthermore, multi-image posts (like those in the new MMSD3.0 benchmark) add a layer of complexity: which image holds the "punchline," and how should we weigh them against the text?

Methods

The authors propose SEAF, a dual-branch architecture designed to handle these challenges through "selective enrichment":

  1. Semantic Evidence Branch: This branch focuses on meaning. It combines the original post text with Optical Character Recognition (OCR) text from the images and Entity Background generated by a frozen MLLM. Crucially, it only uses the "background" (e.g., "Who is this person?") to enrich the text, avoiding redundant visual descriptions.
  2. Aligned Visual Evidence Branch: This branch focuses on vision and relationships. It uses global and local image features but incorporates MLLM-generated image descriptions solely to calculate "alignment statistics." This helps the model understand how well the text matches the visual content without cluttering the semantic sequence.
  3. Text-Anchored Smoothed Aggregation (TASA): For posts with multiple images, SEAF uses TASA to weight each image's contribution. Unlike standard attention that might ignore images with low literal similarity to the text, TASA "smooths" these weights to ensure that even a visually "unrelated" image—which might contain the sarcastic twist—is still considered.
  4. Sarcasm-aware Contrastive Regularization (SaCR): A specialized training loss that encourages the model to map non-sarcastic text-image pairs closer together and sarcastic (incongruent) pairs further apart in a shared feature space.
Figure 2: SEAF architecture. The frozen multimodal large language model (MLLM) supplies neutral entity background and image descriptions for semantic enrichment and visual alignment, respectively. The two branches fuse classification scores for sarcasm prediction.

Datasets

The researchers evaluated SEAF on three key benchmarks:

  • MMSD3.0: A recent multi-image dataset (2-4 images per post) which serves as the primary testing ground.
  • MMSD2.0: A standard single-image multimodal sarcasm benchmark.
  • SarcNet: Used to test "cross-dataset transfer," seeing how well a model trained on one dataset performs on a completely different one (Chinese-English) without further tuning.

Key Insights and Ideas

  • Role-Based Information Allocation: The "aha moment" here is that not all MLLM output is created equal. Treating entity background as text and image descriptions as alignment data prevents the model from getting confused by redundant descriptions of things it can already "see" in the pixel data.
  • Context vs. Description: Figure 1 illustrates this perfectly. Knowing who Hunter Biden is (background) is vital for the first example, while reading the "Rules of the Internet" (OCR) is the only way to understand the second. SEAF is built to capture both.
  • The "Punchline" Problem: In multi-image sarcasm, the most important image is often the one that doesn't match the text. TASA’s smoothing mechanism ensures these "weakly aligned" but high-signal images aren't filtered out by the model's attention.
Figure 1: Two sarcastic MMSD3.0 posts: background adds missing context (top), OCR reveals image text (bot- tom), and generated descriptions overlap with visible content. Labels (a) and (b) denote the two images within each post.

Key Results

SEAF set a new standard across the board:

  • MMSD3.0: Achieved a 86.17% Macro-F1, outperforming the previous state-of-the-art (CIRM) by over 3.6 points.
  • MMSD2.0: Reached 85.43% Macro-F1, proving the architecture is just as effective for single-image posts.
  • Ablation Studies: The authors found that removing the "Entity Background" dropped performance by 2.52 points, while removing the entire visual branch caused a massive 3.05-point dip, confirming that both branches are essential.
Figure 3: MMSD3.0 ablation changes in macro-F1 (vari- ant minus full SEAF; points). D–I: description–image alignment. Full results: Appendix D.

Limitations and Future Work

  • MLLM Dependency: The system's performance is tied to the quality of the frozen MLLM. If the MLLM provides inaccurate background info, the sarcasm detection may fail.
  • Language and Culture: While the authors tested transfer to SarcNet, the core training remains English-centric. Sarcasm is deeply cultural, and future work could explore more diverse linguistic and regional contexts.
  • Metadata: The current model doesn't utilize other potential cues like user history or product ratings, which are often available in real-world social media or e-commerce settings.

Summary updated

Abstract

Multimodal sarcasm detection requires textual, visual, and contextual evidence, particularly when multiple images provide complementary cues. Multimodal large language models (MLLMs) can generate auxiliary information from a post's text and images, but their outputs may mix useful background knowledge with redundant visual descriptions. We propose SEAF, a dual-branch framework for selective semantic enrichment and visual evidence fusion. SEAF assigns generated fields to distinct roles: entity background enriches semantic input, while image descriptions supply alignment statistics. The semantic branch combines the original post text, background from a frozen MLLM, and optical character recognition (OCR) text. The aligned visual branch combines global and local visual features with statistics of cross-modal and inter-image similarity. Text-Anchored Smoothed Aggregation (TASA) weights images by their similarity to the original post text while preserving contributions from images with low literal similarity. Summing the branch classification scores integrates contextual semantics with visual evidence. Experiments on the multi-image MMSD3.0 benchmark show that SEAF outperforms all compared methods.
BibTeX
@misc{wan2026seaf,
  author = {Faxian Wan and Xiaocui Yang and Yifan Cao and Shi Feng and Daling Wang and Yifei Zhang},
  title = {{SEAF: Selective Semantic Enrichment and Visual Evidence Fusion for Multimodal Sarcasm Detection}},
  year = {2026},
  eprint = {2611.25672},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25672},
}
Figure