Introduction
Multimodal Stance Detection (MSD) is essential for interpreting public sentiment on social media by analyzing both text and visual data. Traditional approaches primarily focus on "learning to fuse"—using mechanisms like cross-attention to combine features from different modalities. However, these methods often fall into "semantic traps" like irony or sarcasm, where the literal meaning of text contradicts the visual context. To address these limitations, we propose MIND (Meta-cognitive Intuitive-reflective Network for Dual-reasoning), shifting the focus from simple feature fusion to explicit cognitive reasoning.
The Challenge of Semantic Traps
Social media users frequently employ complex rhetorical devices. For instance, a post might use positive language ("Oh look how clever!") alongside a grim image (e.g., coffins) to express a negative stance. Standard fusion models often prioritize textual keywords, leading to misclassification. These models lack the high-level reasoning necessary to resolve inter-modal conflicts and logical inconsistencies.
Methodology: The MIND Framework
Inspired by the dual-process theory of human cognition, MIND implements a self-improving loop consisting of two primary stages:
- Multimodal Knowledge Perception: The system extracts entities and generates captions using a Multimodal Large Language Model (MLLM). It utilizes the Segment Anything Model (SAM) to ground textual entities in specific image regions, ensuring precise visual-semantic alignment.
- Experience-Driven Intuitive Reasoning (System 1): This stage mimics rapid, subconscious judgment. The model queries "Experience Pools" for Modality and Semantics to retrieve relevant past reasoning patterns, forming an initial hypothesis.
- Meta-cognitive Reflective Reasoning (System 2): This "slow thinking" stage employs Modality-CoT and Semantic-CoT (Chain-of-Thought) to scrutinize the initial judgment. It identifies potential biases or shallow interpretations and formulates an adaptive strategy for the final decision.
Experimental Results
MIND was evaluated on the MMSD benchmark, covering diverse domains such as COVID-19 discourse, the Russo-Ukrainian Conflict, and US elections.
- Superior Performance: Built on a Qwen-2.5-VL-32B backbone, MIND outperformed state-of-the-art baselines, including GPT-4V. On the MTSE (Election) dataset, it achieved a Macro-F1 score of 70.87%.
- Robust Generalization: In zero-shot scenarios with unseen targets, MIND maintained high accuracy (e.g., 73.78% on MTSE), whereas traditional models saw significant performance degradation. This demonstrates that reasoning-based strategies are more transferable than simple pattern matching.
- Ablation Insights: Removing the Experience Pools resulted in the most significant performance drop, highlighting the importance of recalling and refining past reasoning traces.
Conclusion and Future Work
MIND demonstrates that explicit reasoning is superior to feature fusion for complex multimodal tasks. By mimicking human cognitive processes, it effectively navigates the nuances of social media communication. While the current framework faces computational overhead due to its multi-stage reasoning, future research will focus on knowledge distillation to create more efficient, real-time versions of the dual-reasoning system.
Summary updated