Sentiment-enhanced Multi-hop Connected Graph Attention Network for Multimodal Aspect-Based Sentiment Analysis

Linlin Zhu, Heli Sun, Xiaoyong Huang, Qi Zhang, Ruichen Cao, Liang HeIJCAI 2025 · DOI

Introduction

Multimodal Aspect-Based Sentiment Analysis (MABSA) is a sophisticated task in natural language processing that requires identifying specific entities (aspects) in a post containing both text and images and determining the sentiment associated with each. While simple sentiment analysis might suffice for straightforward statements, real-world social media content often involves complex grammatical structures, multiple subjects, and conflicting visual-textual cues.

Researchers from Xi’an Jiaotong University have addressed these challenges with the Sentiment-enhanced Multi-hop Connected Graph Attention Network (MCG). This model moves beyond simple word-image pairing by mapping the "structural importance" of a sentence through its syntactic dependencies to accurately resolve sentiment ambiguity.

Overview of the MABSA task, illustrating textual dependencies, visual regions, and the resulting sentiment tree.

The Challenge: Syntactic Blindness and Sentiment Complexity

Current MABSA research faces two primary hurdles. First, sentiment complexity arises in posts where multiple aspects are mentioned with differing sentiments, often exacerbated by limited training data. Second, syntactic blindness occurs when models fail to distinguish the relative importance of grammatical connections. In a sentence like "Barcelona Crowned 2016 La Liga Champions," the word "Champions" serves as a structural "hub." Standard models often treat all syntactic links equally or rely on linear word distance, missing the semantic "gravity" of these central hub words.

Proposed Methodology: The MCG Model

The MCG architecture introduces three key innovations to bridge these gaps:

1. Sentiment-enhanced Pre-training

To mitigate data sparsity, the authors implemented a contrastive pre-training task. The model is trained to distinguish between "positive" samples (correct text, aspect, and label) and "negative" samples (correct text and aspect, but an incorrect random label). This process sharpens the model’s ability to detect subtle sentiment nuances before fine-tuning on specific tasks.

2. Multi-hop Connected Syntactic Dependency Graph

Rather than limiting analysis to immediate neighbors in a dependency tree, MCG utilizes "multi-hop" connectivity. By calculating powers of the adjacency matrix ($M^k$), the model identifies words that are highly connected across the entire sentence structure. These "hubs" are prioritized as they typically carry the most significant semantic weight for sentiment determination.

3. Multi-hop Connected Graph Attention (MCGA)

The MCGA mechanism serves as the fusion point for visual features (extracted via Vision Transformer, ViT) and textual features (extracted via RoBERTa). Guided by the multi-hop syntactic graph, the attention mechanism is forced to focus on the key syntactic relations that actually drive the overall sentiment of the aspect.

The MCG architecture, featuring sentiment-enhanced pre-training and the multi-hop connected graph attention mechanism.

Experimental Results

The MCG model was evaluated on the standard Twitter-2015 and Twitter-2017 benchmarks across three sub-tasks: Joint Analysis (JMASA), Aspect Extraction (MATE), and Sentiment Classification (MASC).

  • State-of-the-Art Performance: On the Twitter-2017 JMASA task, MCG achieved an F1 score of 77.1%, significantly outperforming the previous state-of-the-art (AESAL at 72.0%).
  • Superiority over LLMs: When compared against large language models like Llama-3.2 and GLM-4V, MCG demonstrated a clear advantage. In the JMASA task on Twitter-2015, MCG's F1 score of 70.9% nearly doubled that of Llama-3.2 (44.3%), suggesting that specialized architectures sensitive to syntax are more effective for fine-grained multimodal tasks than general-purpose LLMs.
  • Ablation Insights: Studies confirmed that visual features are critical; removing image data led to a significant performance drop, proving that visual context is essential for resolving textual ambiguity.

Conclusion and Future Work

The MCG model demonstrates that integrating multi-hop syntactic connectivity with sentiment-enhanced pre-training provides a robust framework for multimodal sentiment analysis. While the model currently relies on external dependency parsers (like spaCy), which may struggle with informal social media text, its ability to identify structural "hubs" marks a significant advancement. Future research will focus on making syntactic learning more adaptive to diverse dialects and exploring more granular sentiment expressions.

Summary updated

BibTeX
@inproceedings{zhu2025sentiment,
  author = {Linlin Zhu and Heli Sun and Xiaoyong Huang and Qi Zhang and Ruichen Cao and Liang He},
  title = {{Sentiment-enhanced Multi-hop Connected Graph Attention Network for Multimodal Aspect-Based Sentiment Analysis}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/933},
}
Figure