TL;DR
Multimodal sarcasm detection is notoriously difficult for monolithic large language models (LLMs) because it requires resolving complex contradictions between text and images. To solve this, Commander-GPT introduces a multi-agent routing framework inspired by military command structures that decomposes sarcasm detection into specialized sub-tasks (like facial expression recognition and sentiment analysis). This mixture-of-experts approach yields up to an 11.7% average F1-score improvement over state-of-the-art baselines.
What Problem Does This Paper Address?
Sarcasm is a sophisticated linguistic phenomenon where the intended meaning diverges sharply from the literal words (e.g., saying "Oh great, another meeting" when frustrated). In multimodal settings (text + images), detecting sarcasm requires aligning subtle linguistic cues with visual context, facial expressions, or embedded scene text.
While state-of-the-art LLMs excel at standard NLP tasks, growing evidence shows they struggle significantly with sarcasm, often performing close to random guessing. The authors identify a fundamental limitation in current methods: they treat sarcasm detection as an undivided, end-to-end task. Monolithic models try to implicitly execute multiple layers of reasoning simultaneously, failing to explicitly isolate and weigh conflicting cues—such as a positive text paired with an annoyed facial expression.
Methods
The authors propose Commander-GPT, a structured multi-agent framework that replaces a single monolithic model with a "right person for the right job" command hierarchy.
The architecture operates in three distinct stages:
- Agent Routing: A central "commander" uses a routing scorer to analyze the input and dynamically activate only the sub-task agents necessary for that specific sample, optimizing computational efficiency.
- Subtask Execution: The problem is broken down into six cognitively meaningful sub-tasks, each handled by a specialized expert model:
- Context Modeling: Analyzes broader conversational context (handled by Llama 3-8B).
- Sentiment Analysis: Extracts fine-grained emotional polarity from text (handled by RoBERTa).
- Rhetorical Device Recognition: Identifies linguistic patterns like irony or hyperbole (handled by GLM-2B).
- Facial Expression Recognition: Extracts facial emotions from images (handled by ViT-FER).
- Image Summarization: Generates comprehensive scene descriptions (handled by BLIP-2).
- Scene Text Recognition: Extracts text embedded within images (handled by OCR-2.0).
- Result Integration: The commander model aggregates the intermediate outputs from all activated agents to make the final binary sarcasm judgment.
The authors evaluated three types of centralized commanders: a lightweight encoder-based model (BERT+ViT), four medium-sized open-source autoregressive LLMs (e.g., DeepSeek-VL, Qwen-VL), and two large proprietary LLMs (GPT-4o and Gemini Pro).
Datasets
The framework was evaluated on two primary multimodal sarcasm detection benchmarks collected from Twitter:
- MMSD: A standard benchmark consisting of paired textual tweets and visual data with ground-truth sarcasm annotations.
- MMSD 2.0: An enhanced, more reliable version of MMSD featuring improved text-image alignment and more diverse visual content to enforce stricter multi-modal reasoning.
- SemEval 2018 Task 3: Used as an out-of-distribution (OOD) dataset to evaluate the model's robustness and generalization capabilities across new domains.
Key Insights and Ideas
- Decomposition Beats Scale: Instead of relying on the raw scale of a single massive model, explicitly breaking the problem into modular, specialized sub-tasks allows smaller models to collectively outperform monolithic giants.
- The Power of Fine-Tuned Small Commanders: Surprisingly, a lightweight, fine-tuned BERT+ViT model serving as the commander achieved the highest absolute F1-scores (86.7 on MMSD, 85.8 on MMSD 2.0), outperforming zero-shot giant models like GPT-4o. Task-specific adaptation can trump raw model scale for nuanced pragmatic tasks.
- Dynamic Routing Prevents Noise: Not every sample contains scene text or visible facial expressions. By conditionally invoking agents via a routing scorer, the framework avoids introducing irrelevant noise into the final decision-making process.
Key Results
- Consistent Baseline Performance Gains: Commander-GPT consistently outperformed five state-of-the-art prompting strategies, including Chain-of-Thought (CoT) and Plan-and-Solve.
- Quantitative Improvements: On average, the framework achieved a 4.4% F1-score improvement on MMSD and an 11.7% F1-score improvement on MMSD 2.0.
- Massive Gains for Smaller Models: The routing architecture was highly effective at compensating for the individual limitations of smaller open-source models. For instance, using MiniCPM-V2 as a commander resulted in an absolute F1-score jump of 22.8% on MMSD 2.0 compared to its baseline.
Limitations and Future Work
- Dependency on Agent Quality: The framework's accuracy is tightly bound to the performance of its underlying specialist agents; limited agent expressiveness or errors in sub-task outputs directly degrade the commander's final decision.
- Routing Data Overhead: Training the learnable routing classifier requires labeled routing supervision instances, which the authors had to distill from GPT-4o.
- Computational Complexity: Deploying and running multiple specialized models in parallel—even when selectively routed—incurs higher infrastructure and latency overhead than an end-to-end monolithic model.
- Future Directions: The authors suggest exploring automatic subtask discovery and more efficient routing mechanisms to optimize the framework.
Summary updated