TL;DR
The GRASP framework advances Multimodal Sarcasm Target Identification (MSTI) by combining explicit visual grounding with structured Chain-of-Thought (CoT) reasoning. By utilizing a dual-stage optimization process—comprising supervised fine-tuning with a coordinate-aware loss and reinforcement learning with fine-grained rewards—GRASP accurately localizes both the textual and visual triggers of sarcasm while providing human-interpretable explanations.
What Problem Does This Paper Address?
Traditional Multimodal Sarcasm Detection (MSD) treats sarcasm as a simple binary classification problem: is an image-text pair sarcastic or not? However, this macro-level approach acts as a black box, failing to explain why something is sarcastic or where the sarcastic triggers lie.
To solve this, researchers recently introduced Multimodal Sarcasm Target Identification (MSTI). MSTI requires models to pinpoint the exact text spans and visual regions that create the sarcastic incongruity. For instance, as shown in Figure 1, a model must connect the sarcastic phrase "soooo expected. not" with a bounding box around a perilously low-flying airplane.
Existing MSTI methods suffer from two major flaws:
- Lack of Interpretability: They rely on implicit feature fusion or complex multi-stage pipelines that directly output coordinates and labels without explaining the logical trajectory.
- Information Loss: Many approaches compress complex visual signals into text captions or tags before reasoning, which discards fine-grained spatial details crucial for detecting subtle visual anomalies.
Methods
The authors introduce GRASP (Grounded Chain-of-Thought ReAsoning with Dual-Stage Optimization), an end-to-end framework that enforces structured "Thinking with Images."
The training pipeline consists of two main stages:
- Grounded Chain-of-Thought (CoT) & Supervised Fine-Tuning (SFT): Instead of jumping straight to a label, the model is trained to generate a structured reasoning path inside a
<Think>module. This path includes a Visual Description (listing key objects and their normalized coordinates), a Text Analysis (decoding tone), and a Consistency Check (evaluating cross-modal contradictions). Crucially, the authors implement a Coordinate-aware Weighted Loss during SFT, heavily penalizing errors on bounding box tokens ($\lambda_{coord} = 10$) to force the model to prioritize spatial precision. - Fine-Grained Target Policy Optimization (FTPO): To align the reasoning chain with the final outputs, the authors introduce an outcome-supervised reinforcement learning stage based on Group Relative Policy Optimization (GRPO). FTPO samples multiple candidate outputs and scores them using a multi-dimensional reward function covering format compliance ($R_{fmt}$), semantic accuracy ($R_{acc}$), visual grounding IoU ($R_{box}$), text exact match ($R_{txt}$), and a penalty for redundant generation ($P_{over}$).
Datasets
The authors discovered that the existing benchmark, MSTI 2.0, suffered from extreme class imbalance (only 75 non-sarcastic samples out of over 5,000) and lacked bounding box annotations for many salient targets.
To resolve this, they curated MSTI-MAX. They augmented the dataset with high-quality non-sarcastic pairs from MMSD and MMSD 2.0 and manually corrected missing visual annotations using domain experts. The final dataset contains 9,015 samples with a balanced distribution of sarcastic and non-sarcastic instances across different modalities.
Key Insights and Ideas
- "Thinking with Images" Preserves Detail: By forcing the model to output normalized coordinates during its text reasoning phase, GRASP treats the visual space as an active cognitive workspace rather than compressing it into static captions beforehand.
- The Breakdown of the Scaling Law: Interestingly, simply scaling up model parameters does not guarantee better performance on fine-grained MSTI tasks. Massive proprietary models like Gemini-3-Pro excel at binary classification but struggle significantly with zero-shot visual target localization compared to smaller, vision-optimized models like Qwen3-VL.
- Asymmetric Gradient Feedback is Necessary: Standard cross-entropy treats all tokens equally. Amplifying the loss specifically for coordinate tokens prevents the model from taking text-only shortcuts and forces it to resolve cross-modal incongruities.
Key Results
GRASP achieves state-of-the-art (SOTA) performance in visual sarcasm target localization while remaining highly competitive in text target identification and binary detection.
- Visual Grounding Performance: On the MSTI-MAX dataset, GRASP (built on Qwen3-VL-8B) achieved a Visual Target Average Precision (AP) of 46.71, drastically outperforming the previous SOTA task-specific model, CofiPara, which scored only 9.79.
- Reasoning Quality (LLM-as-a-Judge): Because standard metrics only evaluate final correctness, the authors pioneered an LLM-as-a-Judge evaluation using Qwen3.5-Plus to score the intermediate reasoning chains from 1 to 5. GRASP outperformed or matched massive models like Gemini-3-Pro in Visual Perception and Logical Consistency.
Limitations and Future Work
The authors note that despite GRASP's success, fine-grained multimodal sarcasm identification remains constrained by high computational overhead during reinforcement learning. They suggest that future work should focus on exploring more efficient training paradigms and incorporating broader contextual modeling to capture highly nuanced, culturally dependent sarcastic expressions.
Summary updated