Introduction
Multimodal Aspect-Based Sentiment Analysis (MABSA) is a critical task in social media processing, requiring models to identify specific aspects (e.g., entities or brands) and determine the sentiment directed toward them using both textual and visual cues. Despite advancements, existing models often struggle with two primary issues: weak cross-modal correlation—where the model fails to link text terms to visual regions—and syntactic "flatness," where grammatical nuances are overlooked in favor of simple word proximity.
The AESAL Framework
To address these limitations, the Aspect Enhancement and Syntactic Adaptive Learning (AESAL) framework introduces a multi-pronged approach to multimodal reasoning. The model utilizes RoBERTa for text encoding and a Vision Transformer (ViT) for image encoding to ensure high-quality feature extraction.
Aspect Enhancement (AE)
AESAL utilizes a specialized pre-training task designed to "anchor" aspect terms to their corresponding visual regions. By training on positive pairs (original text-image) and negative pairs (where aspect terms are masked), the model learns to specifically look for the visual representation of an aspect, ensuring the image provides context rather than noise. This prevents the model from looking at the image generally and forces it to focus on the objects that represent the aspect terms.
Syntactic Adaptive Learning (SAL)
Rather than treating all words in a sentence as equally connected, AESAL constructs multiple syntactic dependency graphs based on "hops" in a grammar tree. This allows the model to capture local modifiers and global sentence structure separately. These graphs guide a Multi-Channel Adaptive Graph Convolutional Network (GCN) that processes text-to-text, text-to-image, and image-to-text interactions, preventing "modality collapse" where one input dominates the other.
Experimental Results
The framework was evaluated on the Twitter-2015 and Twitter-2017 benchmarks. The latter is particularly challenging, with 66.46% of sentences containing multiple aspects. AESAL achieved state-of-the-art results across several subtasks:
- Joint Task (JMASA): F1 scores of 69.5% (2015) and 72.0% (2017).
- Aspect Term Extraction (MATE): A significant 94.7% F1 score on the 2017 dataset.
- Sentiment Classification (MASC): A 2.4% accuracy improvement over previous baselines on the 2017 dataset.
Ablation studies confirmed that both the visual input and the aspect-enhancement pre-training are vital for performance, particularly in complex sentences with multiple aspects. While the model showed slight sensitivity to dataset size in the 2015 sentiment classification task, its overall performance remains superior to previous baselines like VLP-MABSA and CMMT.
Conclusion
AESAL demonstrates that effective MABSA requires more than just concatenating modalities; it requires a deep understanding of how syntax structures sentiment and how specific words ground into visual regions. While text remains the primary driver of sentiment, the adaptive use of images as auxiliary context provides the necessary edge for high-accuracy social media analysis.
Summary updated