TL;DR
Multimodal Large Language Models (MLLMs) frequently "cheat" when interpreting metaphors, relying on superficial textual shortcuts rather than true cross-modal reasoning. To address this, the authors introduce M³R-Bench, a unified benchmark requiring evidence-grounded explanations, and M³R-Reasoner, an 8B model that leverages curriculum learning and reinforcement learning to outperform proprietary giants like GPT-5.5 and Claude-Sonnet-4.6.
What Problem Does This Paper Address?
Metaphors allow us to understand abstract concepts ("Targets") by mapping them to concrete domains ("Sources"). In multimodal settings like advertising or social media, this mapping relies on a tight interplay between images and text.
However, existing benchmarks evaluate metaphor detection, target-source mapping, and sentiment analysis as isolated subtasks. This fragmented approach creates an evidence-mapping mismatch: models can easily exploit "textual shortcuts"—guessing the correct metaphor or sentiment solely from a catchy text slogan while completely ignoring the visual evidence.
To achieve true multimodal understanding, models must not only output the correct labels but also provide verifiable, evidence-grounded explanations of how the visual and textual cues construct the metaphor.
Methods
The authors tackle this challenge with a two-part contribution: a rigorous evaluation framework and a novel reasoning model.
1. M³R-Bench Construction
The benchmark features a stage-wise explanation chain following a strict logical flow: $$\text{Evidence Identification} \rightarrow \text{Mapping Establishment} \rightarrow \text{Sentiment Inference}$$ Annotations were generated via an MLLM-led, human-assisted protocol utilizing five state-of-the-art models (including GPT-5.5 and Claude-Opus-4.7) followed by rigorous human adjudication and verification.
2. M³R-Reasoner
To fix the reasoning flaws found in standard MLLMs, the authors developed M³R-Reasoner using a Qwen3-VL-8B-Instruct backbone through two main stages:
- Curriculum-Based Supervised Fine-Tuning (SFT): The model is trained sequentially on three corpora of increasing task complexity. It starts with simple metaphor detection and justification, progresses to target-source mapping, and finishes with full-task understanding (including sentiment inference).
- Task-Aware Reinforcement Learning: Using Group Relative Policy Optimization (GRPO), the model is optimized against a composite reward function. This function scores format compliance, exact label matching, and semantic alignment of target/source concepts using BERTScore. A frozen reference policy provides KL-regularization to ensure the model's reasoning stays anchored to the visual evidence.
Datasets
The paper introduces M³R-Bench, a newly curated dataset containing 1,000 image-text instances (510 metaphorical, 490 non-metaphorical). The data is balanced across three distinct real-world sources:
- Social Media (Twitter and Facebook posts)
- Commercial Advertisements
- Public-Service Advertisements
Each instance includes rich, human-verified annotations for metaphor presence, core target-source pairs, sentiment, visual evidence rationales, and sentiment justifications.
Key Insights and Ideas
- The Illusion of Performance: Standard MLLMs are highly prone to Textual Shortcut Bias (accounting for 43% of GPT-4o's errors). They routinely output correct sentiment and metaphor labels based on text alone while hallucinating or misidentifying the underlying visual structure.
- Granularity Mismatch: When models fail to map metaphors correctly, they tend to output broad, theme-level concepts (e.g., "environmental protection") rather than the specific entities involved in the cross-domain mapping (e.g., mapping a "solar farm" to a "flowing river").
- RL for Reasoning Alignment: Applying GRPO with dense, structured rewards effectively forces the model's internal
<think>tokens to align with the final structured<label>outputs, significantly mitigating the evidence-mapping mismatch.
Key Results
Evaluated on the unified full-task setting of M³R-Bench, the 8B M³R-Reasoner consistently outperformed much larger open-weight and proprietary models:
- Label Accuracy: M³R-Reasoner achieved 72.53% Macro-F1 on Metaphor Detection and 71.10% on Sentiment Classification, substantially outperforming GPT-5.5 (54.56% / 61.77%) and Claude-Sonnet-4.6 (39.03% / 57.11%).
- Explanation Quality: On rubric-based human-aligned evaluations, M³R-Reasoner improved Visual Evidence and Sentiment Justification scores over GPT-5.5 by 28.45 and 30.11 points, respectively. It also surpassed Claude-Sonnet-4.6 by 8.00 points in mean rubric score.
Limitations and Future Work
- Visual Grounding Bottlenecks: While M³R-Reasoner significantly reduced textual shortcut errors, visual grounding failures (such as interpreting complex image manipulations like morphological fusion or symbolic transformations) remain the primary bottleneck, representing 15% of its remaining errors.
- Persistent Hallucinations: The rate of completely hallucinated reasoning remained unchanged during reinforcement learning, indicating that underlying vision-language backbones still possess fundamental limitations in cross-modal alignment that RL alone cannot fix. The authors suggest deeper visual-textual pre-training as a promising future direction.
Summary updated