GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification

Faxian Wan, Xiaocui Yang, Yifan Cao, Shi Feng, Daling Wang, Yifei ZhangarXiv 2026 · arXiv

TL;DR

The GRASP framework advances Multimodal Sarcasm Target Identification (MSTI) by combining explicit visual grounding with structured Chain-of-Thought (CoT) reasoning. By utilizing a dual-stage optimization process—comprising supervised fine-tuning with a coordinate-aware loss and reinforcement learning with fine-grained rewards—GRASP accurately localizes both the textual and visual triggers of sarcasm while providing human-interpretable explanations.

What Problem Does This Paper Address?

Traditional Multimodal Sarcasm Detection (MSD) treats sarcasm as a simple binary classification problem: is an image-text pair sarcastic or not? However, this macro-level approach acts as a black box, failing to explain why something is sarcastic or where the sarcastic triggers lie.

To solve this, researchers recently introduced Multimodal Sarcasm Target Identification (MSTI). MSTI requires models to pinpoint the exact text spans and visual regions that create the sarcastic incongruity. For instance, as shown in Figure 1, a model must connect the sarcastic phrase "soooo expected. not" with a bounding box around a perilously low-flying airplane.

Figure 1: An illustrative example of MSTI. The sarcasm arises from the contradiction between the specific textual span “soooo expected. not” and the localized visual region (de- picting a perilously low-flying airplane), which collectively conveys the sarcastic intent. Our GRASP framework not only outputs fine-grained multimodal targets directly but also yields explicit grounding chain-of-thought reasoning.

Existing MSTI methods suffer from two major flaws:

  1. Lack of Interpretability: They rely on implicit feature fusion or complex multi-stage pipelines that directly output coordinates and labels without explaining the logical trajectory.
  2. Information Loss: Many approaches compress complex visual signals into text captions or tags before reasoning, which discards fine-grained spatial details crucial for detecting subtle visual anomalies.

Methods

The authors introduce GRASP (Grounded Chain-of-Thought ReAsoning with Dual-Stage Optimization), an end-to-end framework that enforces structured "Thinking with Images."

The training pipeline consists of two main stages:

  1. Grounded Chain-of-Thought (CoT) & Supervised Fine-Tuning (SFT): Instead of jumping straight to a label, the model is trained to generate a structured reasoning path inside a <Think> module. This path includes a Visual Description (listing key objects and their normalized coordinates), a Text Analysis (decoding tone), and a Consistency Check (evaluating cross-modal contradictions). Crucially, the authors implement a Coordinate-aware Weighted Loss during SFT, heavily penalizing errors on bounding box tokens ($\lambda_{coord} = 10$) to force the model to prioritize spatial precision.
  2. Fine-Grained Target Policy Optimization (FTPO): To align the reasoning chain with the final outputs, the authors introduce an outcome-supervised reinforcement learning stage based on Group Relative Policy Optimization (GRPO). FTPO samples multiple candidate outputs and scores them using a multi-dimensional reward function covering format compliance ($R_{fmt}$), semantic accuracy ($R_{acc}$), visual grounding IoU ($R_{box}$), text exact match ($R_{txt}$), and a penalty for redundant generation ($P_{over}$).
Figure 3: Overview of the GRASP training pipeline. The framework consists of two stages: (1) Supervised Fine-Tuning (SFT) with coordinate-aware weighted loss employing structured grounded CoT reasoning; (2) Fine-Grained Target Policy Optimization (FTPO) with multidimensional rewards, which further strengthens reasoning quality and visual grounding precision.

Datasets

The authors discovered that the existing benchmark, MSTI 2.0, suffered from extreme class imbalance (only 75 non-sarcastic samples out of over 5,000) and lacked bounding box annotations for many salient targets.

To resolve this, they curated MSTI-MAX. They augmented the dataset with high-quality non-sarcastic pairs from MMSD and MMSD 2.0 and manually corrected missing visual annotations using domain experts. The final dataset contains 9,015 samples with a balanced distribution of sarcastic and non-sarcastic instances across different modalities.

Comparison of target distributions between MSTI 2.0 and MSTI-MAX

Key Insights and Ideas

  • "Thinking with Images" Preserves Detail: By forcing the model to output normalized coordinates during its text reasoning phase, GRASP treats the visual space as an active cognitive workspace rather than compressing it into static captions beforehand.
  • The Breakdown of the Scaling Law: Interestingly, simply scaling up model parameters does not guarantee better performance on fine-grained MSTI tasks. Massive proprietary models like Gemini-3-Pro excel at binary classification but struggle significantly with zero-shot visual target localization compared to smaller, vision-optimized models like Qwen3-VL.
  • Asymmetric Gradient Feedback is Necessary: Standard cross-entropy treats all tokens equally. Amplifying the loss specifically for coordinate tokens prevents the model from taking text-only shortcuts and forces it to resolve cross-modal incongruities.

Key Results

GRASP achieves state-of-the-art (SOTA) performance in visual sarcasm target localization while remaining highly competitive in text target identification and binary detection.

  • Visual Grounding Performance: On the MSTI-MAX dataset, GRASP (built on Qwen3-VL-8B) achieved a Visual Target Average Precision (AP) of 46.71, drastically outperforming the previous SOTA task-specific model, CofiPara, which scored only 9.79.
  • Reasoning Quality (LLM-as-a-Judge): Because standard metrics only evaluate final correctness, the authors pioneered an LLM-as-a-Judge evaluation using Qwen3.5-Plus to score the intermediate reasoning chains from 1 to 5. GRASP outperformed or matched massive models like Gemini-3-Pro in Visual Perception and Logical Consistency.
Figure 4: LLM-as-a-Judge evaluation results. Higher values indicate better performance on a 1–5 scale.

Limitations and Future Work

The authors note that despite GRASP's success, fine-grained multimodal sarcasm identification remains constrained by high computational overhead during reinforcement learning. They suggest that future work should focus on exploring more efficient training paradigms and incorporating broader contextual modeling to capture highly nuanced, culturally dependent sarcastic expressions.

Summary updated

Abstract

Moving beyond the traditional binary classification paradigm of Multimodal Sarcasm Detection, Multimodal Sarcasm Target Identification (MSTI) presents a more formidable challenge, requiring precise localization of fine-grained targets such as textual phrases and visual regions. Existing approaches predominantly rely on implicit cross-modal alignment, offering limited interpretability and suboptimal fine-grained localization. To address these limitations, we propose GRASP, Grounded Chain-of-Thought ReAsoning with Dual-Stage Optimization for Multimodal Sarcasm Prediction and Target Identification, a framework that integrates visual grounding with explicit Chain-of-Thought (CoT) reasoning to move beyond black-box MSTI. Specifically, we curate MSTI-MAX, a refined dataset that mitigates class imbalance and enriches multimodal sarcasm cues. We introduce Grounded CoT reasoning, which explicitly anchors sarcasm-related visual regions within the reasoning trajectory and prompts the model to articulate rationales before predicting the final classification labels and sarcasm targets. Furthermore, we employ a dual-stage outcome-supervised joint optimization strategy: Supervised Fine-Tuning with a coordinate-aware weighted loss, followed by Fine-Grained Target Policy Optimization. Extensive experiments demonstrate that GRASP outperforms existing baselines in fine-grained sarcasm target identification across modalities, and an LLM-as-a-Judge evaluation quantitatively measures the quality of internal reasoning chains. Our dataset and source code will be released on GitHub.
BibTeX
@misc{wan2026grasp,
  author = {Faxian Wan and Xiaocui Yang and Yifan Cao and Shi Feng and Daling Wang and Yifei Zhang},
  title = {{GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification}},
  year = {2026},
  eprint = {2604.08879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.08879},
}
Figure