Abstract
Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance on many downstream NLP tasks, growing evidence suggests that they struggle with sarcasm understanding. In this paper, we propose Commander-GPT, a modular decision routing framework inspired by military command theory. Rather than relying on a single LLM's capability, Commander-GPT orchestrates a team of specialized LLM agents where each agent will be selectively assigned to a focused sub-task such as context modeling, sentiment analysis, etc. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment. To coordinate these agents, we introduce three types of centralized commanders: (1) a trained lightweight encoder-based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving as moderately capable commanders (e.g., DeepSeek-VL); (3) two large LLM-based commanders (Gemini Pro and GPT-4o) that perform task routing, output aggregation, and sarcasm decision-making in a zero-shot fashion. We evaluate Commander-GPT on the MMSD and MMSD 2.0 benchmarks, comparing five prompting strategies. Experimental results show that our framework achieves 4.4% and 11.7% improvement in F1 score over state-of-the-art (SoTA) baselines on average, demonstrating its effectiveness.
Introduction
Sarcasm is a sophisticated form of communication that often relies on "cross-modal incongruity"—a mismatch between the literal meaning of words and the visual or situational context. For example, a positive caption like "Love it when my mans on a cleaning spree" might be sincere or sarcastic depending on the accompanying imagery and emojis. While Large Language Models (LLMs) excel at general reasoning, they often fail to capture these subtle contradictions, frequently performing little better than random guessing on complex sarcasm benchmarks.
As illustrated in Figure 1, monolithic models often struggle to integrate textual and visual cues correctly. Even advanced models like Llama 3-8B can misinterpret the tone of a message, whereas the proposed modular approach aims to address these failures by decomposing the reasoning process into specialized, manageable steps.
The Commander-GPT Framework
Inspired by military command theory, we propose Commander-GPT, which replaces a single monolithic model with a structured hierarchy of specialized agents managed by a centralized "Commander."
Modular Decomposition
The task is broken down into six cognitively meaningful sub-tasks:
- Linguistic Analysis: Context Modeling (e.g., Llama 3-8B), Sentiment Analysis (e.g., RoBERTa), and Rhetorical Device Recognition (e.g., GLM-2B).
- Visual Analysis: Facial Expression Recognition (e.g., ViT-FER), Image Summarization (e.g., BLIP-2), and Scene Text Recognition (OCR).
Dynamic Routing and Integration
The Commander unit manages the "mission" through a dynamic routing mechanism. Instead of running every agent for every input, the Commander uses a routing scorer to activate only the relevant specialists. This mimics human cognition—for instance, we only process scene text if text is visually present. Once the activated agents provide their reports, the Commander integrates this "intelligence" to make the final sarcasm judgment.
Experimental Results
We evaluated Commander-GPT on the MMSD and MMSD 2.0 benchmarks. The framework achieved a significant 19.3% relative improvement in F1 score over previous state-of-the-art methods. Specifically, on the MMSD 2.0 benchmark, the framework boosted the F1 score of a standard BERT+ViT setup from 77.2 to 85.8 (+8.6 points).
Ablation studies revealed that Rhetorical Device Recognition and Context Modeling are the most critical components. Removing these modules led to performance drops of up to 29.1% in F1 score, confirming that sarcasm detection is heavily dependent on identifying linguistic subversion and contextual contrast. Furthermore, the framework showed strong generalization on out-of-distribution data from the SemEval 2018 Task 3.
Conclusion
Commander-GPT demonstrates that a modular, routing-based architecture is highly effective for high-order cognitive tasks like multimodal sarcasm detection. By leveraging specialized agents and an intelligent centralized commander, the system significantly outperforms monolithic LLMs. Future work will focus on automatic sub-task discovery and enhancing the robustness of individual agents to handle increasingly ambiguous scenarios.
Summary updated