Introduction
Aspect-Based Multimodal Sentiment Analysis (ABMSA) is a fine-grained task where a model must determine the sentiment toward a specific entity (an "aspect") mentioned in a post containing both text and images. For example, in a tweet about a "new phone" accompanied by a picture of a sunset, the model must recognize that the image is irrelevant to the sentiment regarding the phone.
Most existing models assume that text and images are always semantically related and attempt to fuse them regardless of relevance. This leads to several challenges: semantic mismatch in social media datasets, the "semantic gap" between visual pixels and linguistic sentiments, and regional noise within relevant images.
The TISRI Model Architecture
To address these issues, the Text-Image Semantic Relevance Identification (TISRI) model was developed. TISRI introduces an "image gate" to filter out irrelevant visual information, ensuring that images only influence sentiment predictions when they are semantically relevant to the aspect being discussed.
The model consists of three innovative modules:
Multimodal Feature Relevance Identification (MFRI)
The MFRI module acts as a "gatekeeper" by calculating the similarity between text and image features. If the relevance is low, an Image Gate dynamically reduces the weight of visual information to prevent it from interfering with text-based sentiment analysis.
Image Feature Auxiliary Reconstruction (IFAR)
To bridge the semantic gap, the IFAR module extracts Adjective-Noun Pairs (ANPs) (e.g., "bright screen," "happy face") from images. These ANPs serve as a semantic bridge, helping the model align abstract visual features with concrete linguistic concepts.
Image Feature Filtering (IFF)
During the final fusion stage, a cross-modal attention mechanism focuses on image regions relevant to the text's context, effectively filtering out background noise.
The model utilizes RoBERTa for text encoding and ResNet-152 for image encoding, representing state-of-the-art unimodal feature extraction.
Experimental Evaluation
TISRI was evaluated on two benchmark datasets: TWITTER-2015 and TWITTER-2017. Both datasets include text, associated images, specific aspects, and sentiment labels (positive, negative, or neutral).
Key Results
TISRI achieved state-of-the-art or highly competitive results, outperforming models that rely on image captioning or simple concatenation.
- TWITTER-2015: Accuracy of 78.50% and Macro-F1 of 74.42%.
- TWITTER-2017: Accuracy of 72.53% and Macro-F1 of 71.40%.
Ablation studies confirmed the importance of the new modules. Removing the Image Gate (MFRI) or the semantic bridge (IFAR) led to a significant drop in accuracy (between 0.7% and 2.5%), proving that relevance identification is critical for robust multimodal AI.
Conclusion
The TISRI model demonstrates that knowing when to ignore an image is as important as knowing how to process it. By using ANPs as a semantic bridge and a learnable gate for relevance, the model gains a more intuitive understanding of an image’s emotional content relative to a specific aspect. Future work will explore integrating Large Language Models (LLMs) to further improve semantic alignment and handling more complex multi-class sentiment tasks.
Summary updated