M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding

Hong Jiang, Junnan Zhu, Jingwang Huang, Xiao Sun, Yuming Yang, Jiang Zhong, Ruirui Chen, Jingman Shi, Hao Wu, Nayu Liu, Xinyi Jiang, Kaiwen WeiarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) frequently "cheat" when interpreting metaphors, relying on superficial textual shortcuts rather than true cross-modal reasoning. To address this, the authors introduce M³R-Bench, a unified benchmark requiring evidence-grounded explanations, and M³R-Reasoner, an 8B model that leverages curriculum learning and reinforcement learning to outperform proprietary giants like GPT-5.5 and Claude-Sonnet-4.6.

What Problem Does This Paper Address?

Metaphors allow us to understand abstract concepts ("Targets") by mapping them to concrete domains ("Sources"). In multimodal settings like advertising or social media, this mapping relies on a tight interplay between images and text.

However, existing benchmarks evaluate metaphor detection, target-source mapping, and sentiment analysis as isolated subtasks. This fragmented approach creates an evidence-mapping mismatch: models can easily exploit "textual shortcuts"—guessing the correct metaphor or sentiment solely from a catchy text slogan while completely ignoring the visual evidence.

Figure 1: Motivation of this work. Existing research pre- dicts metaphor and sentiment but misses evidence-grounded Target–Source mappings. In contrast, M3R-Bench jointly evaluates metaphor identification, mapping, and sentiment with evidence-grounded explanations.

To achieve true multimodal understanding, models must not only output the correct labels but also provide verifiable, evidence-grounded explanations of how the visual and textual cues construct the metaphor.

Methods

The authors tackle this challenge with a two-part contribution: a rigorous evaluation framework and a novel reasoning model.

1. M³R-Bench Construction

The benchmark features a stage-wise explanation chain following a strict logical flow: $$\text{Evidence Identification} \rightarrow \text{Mapping Establishment} \rightarrow \text{Sentiment Inference}$$ Annotations were generated via an MLLM-led, human-assisted protocol utilizing five state-of-the-art models (including GPT-5.5 and Claude-Opus-4.7) followed by rigorous human adjudication and verification.

2. M³R-Reasoner

To fix the reasoning flaws found in standard MLLMs, the authors developed M³R-Reasoner using a Qwen3-VL-8B-Instruct backbone through two main stages:

  • Curriculum-Based Supervised Fine-Tuning (SFT): The model is trained sequentially on three corpora of increasing task complexity. It starts with simple metaphor detection and justification, progresses to target-source mapping, and finishes with full-task understanding (including sentiment inference).
  • Task-Aware Reinforcement Learning: Using Group Relative Policy Optimization (GRPO), the model is optimized against a composite reward function. This function scores format compliance, exact label matching, and semantic alignment of target/source concepts using BERTScore. A frozen reference policy provides KL-regularization to ensure the model's reasoning stays anchored to the visual evidence.
Overview of M3R-Reasoner's two-stage training process.

Datasets

The paper introduces M³R-Bench, a newly curated dataset containing 1,000 image-text instances (510 metaphorical, 490 non-metaphorical). The data is balanced across three distinct real-world sources:

  • Social Media (Twitter and Facebook posts)
  • Commercial Advertisements
  • Public-Service Advertisements

Each instance includes rich, human-verified annotations for metaphor presence, core target-source pairs, sentiment, visual evidence rationales, and sentiment justifications.

Figure 2: Data distribution and statistics of M3R-Bench.

Key Insights and Ideas

  • The Illusion of Performance: Standard MLLMs are highly prone to Textual Shortcut Bias (accounting for 43% of GPT-4o's errors). They routinely output correct sentiment and metaphor labels based on text alone while hallucinating or misidentifying the underlying visual structure.
  • Granularity Mismatch: When models fail to map metaphors correctly, they tend to output broad, theme-level concepts (e.g., "environmental protection") rather than the specific entities involved in the cross-domain mapping (e.g., mapping a "solar farm" to a "flowing river").
  • RL for Reasoning Alignment: Applying GRPO with dense, structured rewards effectively forces the model's internal <think> tokens to align with the final structured <label> outputs, significantly mitigating the evidence-mapping mismatch.

Key Results

Evaluated on the unified full-task setting of M³R-Bench, the 8B M³R-Reasoner consistently outperformed much larger open-weight and proprietary models:

  • Label Accuracy: M³R-Reasoner achieved 72.53% Macro-F1 on Metaphor Detection and 71.10% on Sentiment Classification, substantially outperforming GPT-5.5 (54.56% / 61.77%) and Claude-Sonnet-4.6 (39.03% / 57.11%).
  • Explanation Quality: On rubric-based human-aligned evaluations, M³R-Reasoner improved Visual Evidence and Sentiment Justification scores over GPT-5.5 by 28.45 and 30.11 points, respectively. It also surpassed Claude-Sonnet-4.6 by 8.00 points in mean rubric score.

Limitations and Future Work

  • Visual Grounding Bottlenecks: While M³R-Reasoner significantly reduced textual shortcut errors, visual grounding failures (such as interpreting complex image manipulations like morphological fusion or symbolic transformations) remain the primary bottleneck, representing 15% of its remaining errors.
  • Persistent Hallucinations: The rate of completely hallucinated reasoning remained unchanged during reinforcement learning, indicating that underlying vision-language backbones still possess fundamental limitations in cross-modal alignment that RL alone cannot fix. The authors suggest deeper visual-textual pre-training as a promising future direction.

Summary updated

Abstract

Metaphor enables the understanding of abstract concepts through cross-domain mappings while conveying affective attitudes. In multimodal scenarios, visual and textual information jointly construct Target--Source mappings, requiring both conceptual understanding and cross-modal reasoning. However, existing benchmarks mainly evaluate metaphor understanding through isolated subtasks and lack evidence-grounded explanations, making it difficult to assess whether models establish mappings grounded in visual and textual cues.To address these limitations, we introduce M$^3$R-Bench, a unified and evidence-grounded benchmark containing 1,000 image--text instances with human-verified annotations. Guided by Conceptual Metaphor Theory and theories of nonliteral language understanding, M$^3$R-Bench provides joint annotations for metaphor occurrence, Target--Source mapping, sentiment, and stage-wise explanations following ``evidence identification--mapping establishment--sentiment inference.''Evaluations on M$^3$R-Bench reveal that existing models often overlook visual evidence, rely on superficial textual cues, and produce inaccurate Target--Source mappings, exposing a cross-modal evidence--mapping mismatch. To address this mismatch, we propose M$^3$R-Reasoner, which combines curriculum-based reasoning supervision with task-aware reinforcement learning to align model reasoning with metaphor interpretation. Experiments show that, with only an 8B-parameter backbone, M$^3$R-Reasoner outperforms larger proprietary MLLMs across four unified-task metrics and improves Visual Evidence and Sentiment Justification scores over GPT-5.5 by 28.45 and 30.11 points, respectively, while surpassing Claude-Sonnet-4.6 by 8.00 points in mean rubric score. The dataset and code are available at https://github.com/hongshi4/M3R-Bench.
BibTeX
@misc{jiang2026m,
  author = {Hong Jiang and Junnan Zhu and Jingwang Huang and Xiao Sun and Yuming Yang and Jiang Zhong and Ruirui Chen and Jingman Shi and Hao Wu and Nayu Liu and Xinyi Jiang and Kaiwen Wei},
  title = {{M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding}},
  year = {2026},
  eprint = {2608.05817},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.05817},
}
Figure