Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection

Yazhou Zhang, Chunwang Zou, Bo Wang, Jing Qin, Prayag Tiwari

TL;DR

Multimodal sarcasm detection is notoriously difficult for monolithic large language models (LLMs) because it requires resolving complex contradictions between text and images. To solve this, Commander-GPT introduces a multi-agent routing framework inspired by military command structures that decomposes sarcasm detection into specialized sub-tasks (like facial expression recognition and sentiment analysis). This mixture-of-experts approach yields up to an 11.7% average F1-score improvement over state-of-the-art baselines.

What Problem Does This Paper Address?

Sarcasm is a sophisticated linguistic phenomenon where the intended meaning diverges sharply from the literal words (e.g., saying "Oh great, another meeting" when frustrated). In multimodal settings (text + images), detecting sarcasm requires aligning subtle linguistic cues with visual context, facial expressions, or embedded scene text.

While state-of-the-art LLMs excel at standard NLP tasks, growing evidence shows they struggle significantly with sarcasm, often performing close to random guessing. The authors identify a fundamental limitation in current methods: they treat sarcasm detection as an undivided, end-to-end task. Monolithic models try to implicitly execute multiple layers of reasoning simultaneously, failing to explicitly isolate and weigh conflicting cues—such as a positive text paired with an annoyed facial expression.

LLMs struggle with sarcasm detection, often performing near random guessing on complex datasets.

Methods

The authors propose Commander-GPT, a structured multi-agent framework that replaces a single monolithic model with a "right person for the right job" command hierarchy.

The architecture operates in three distinct stages:

  1. Agent Routing: A central "commander" uses a routing scorer to analyze the input and dynamically activate only the sub-task agents necessary for that specific sample, optimizing computational efficiency.
  2. Subtask Execution: The problem is broken down into six cognitively meaningful sub-tasks, each handled by a specialized expert model:
  3. Context Modeling: Analyzes broader conversational context (handled by Llama 3-8B).
  4. Sentiment Analysis: Extracts fine-grained emotional polarity from text (handled by RoBERTa).
  5. Rhetorical Device Recognition: Identifies linguistic patterns like irony or hyperbole (handled by GLM-2B).
  6. Facial Expression Recognition: Extracts facial emotions from images (handled by ViT-FER).
  7. Image Summarization: Generates comprehensive scene descriptions (handled by BLIP-2).
  8. Scene Text Recognition: Extracts text embedded within images (handled by OCR-2.0).
  9. Result Integration: The commander model aggregates the intermediate outputs from all activated agents to make the final binary sarcasm judgment.

The authors evaluated three types of centralized commanders: a lightweight encoder-based model (BERT+ViT), four medium-sized open-source autoregressive LLMs (e.g., DeepSeek-VL, Qwen-VL), and two large proprietary LLMs (GPT-4o and Gemini Pro).

Figure 2: The overall architecture of Commander-GPT.

Datasets

The framework was evaluated on two primary multimodal sarcasm detection benchmarks collected from Twitter:

  • MMSD: A standard benchmark consisting of paired textual tweets and visual data with ground-truth sarcasm annotations.
  • MMSD 2.0: An enhanced, more reliable version of MMSD featuring improved text-image alignment and more diverse visual content to enforce stricter multi-modal reasoning.
  • SemEval 2018 Task 3: Used as an out-of-distribution (OOD) dataset to evaluate the model's robustness and generalization capabilities across new domains.

Key Insights and Ideas

  • Decomposition Beats Scale: Instead of relying on the raw scale of a single massive model, explicitly breaking the problem into modular, specialized sub-tasks allows smaller models to collectively outperform monolithic giants.
  • The Power of Fine-Tuned Small Commanders: Surprisingly, a lightweight, fine-tuned BERT+ViT model serving as the commander achieved the highest absolute F1-scores (86.7 on MMSD, 85.8 on MMSD 2.0), outperforming zero-shot giant models like GPT-4o. Task-specific adaptation can trump raw model scale for nuanced pragmatic tasks.
  • Dynamic Routing Prevents Noise: Not every sample contains scene text or visible facial expressions. By conditionally invoking agents via a routing scorer, the framework avoids introducing irrelevant noise into the final decision-making process.

Key Results

  • Consistent Baseline Performance Gains: Commander-GPT consistently outperformed five state-of-the-art prompting strategies, including Chain-of-Thought (CoT) and Plan-and-Solve.
  • Quantitative Improvements: On average, the framework achieved a 4.4% F1-score improvement on MMSD and an 11.7% F1-score improvement on MMSD 2.0.
  • Massive Gains for Smaller Models: The routing architecture was highly effective at compensating for the individual limitations of smaller open-source models. For instance, using MiniCPM-V2 as a commander resulted in an absolute F1-score jump of 22.8% on MMSD 2.0 compared to its baseline.

Limitations and Future Work

  • Dependency on Agent Quality: The framework's accuracy is tightly bound to the performance of its underlying specialist agents; limited agent expressiveness or errors in sub-task outputs directly degrade the commander's final decision.
  • Routing Data Overhead: Training the learnable routing classifier requires labeled routing supervision instances, which the authors had to distill from GPT-4o.
  • Computational Complexity: Deploying and running multiple specialized models in parallel—even when selectively routed—incurs higher infrastructure and latency overhead than an end-to-end monolithic model.
  • Future Directions: The authors suggest exploring automatic subtask discovery and more efficient routing mechanisms to optimize the framework.

Summary updated

Abstract

Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance on many downstream NLP tasks, growing evidence suggests that they struggle with sarcasm understanding. In this paper, we propose Commander-GPT, a modular decision routing framework inspired by military command theory. Rather than relying on a single LLM's capability, Commander-GPT orchestrates a team of specialized LLM agents where each agent will be selectively assigned to a focused sub-task such as keyword extraction, sentiment analysis, etc. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment. To coordinate these agents, we introduce three types of centralized commanders: (1) a trained lightweight encoder-based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving as moderately capable commanders (e.g., DeepSeek-VL); (3) two large LLM-based commander (Gemini Pro and GPT-4o) that performs task routing, output aggregation, and sarcasm decision-making in a zero-shot fashion. We evaluate Commander-GPT on the MMSD and MMSD 2.0 benchmarks, comparing five prompting strategies. Experimental results show that our framework achieves 4.4% and 11.7% improvement in F1 score over state-of-the-art (SoTA) baselines on average, demonstrating its effectiveness.
BibTeX
@article{zhang2025commander,
  author = {Yazhou Zhang and Chunwang Zou and Bo Wang and Jing Qin and Prayag Tiwari},
  title = {{Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=zRxRbBsqwE},
}
Figure