Abstract
Visual-textual sentiment analysis aims to predict sentiment with the input of a pair of image and text, which poses a challenge in learning effective features for diverse input images. To address this, we propose a holistic method that achieves robust visual-textual sentiment analysis by exploiting a rich set of powerful pre-trained visual and textual prior models. The proposed method consists of four parts: (1) a visual-textual branch to learn features directly from data for sentiment analysis, (2) a visual expert branch with a set of pre-trained "expert" encoders to extract selected semantic visual features, (3) a CLIP branch to implicitly model visual-textual correspondence, and (4) a multimodal feature fusion network based on BERT to fuse multimodal features and make sentiment predictions. Extensive experiments on three datasets show that our method produces better visual-textual sentiment analysis performance than existing methods.
Introduction: The Challenge of Multimodal Sentiment
Predicting sentiment from social media posts is complex because the intended emotion often resides in the subtle interplay between an image and its accompanying text. Images are incredibly diverse; a single dataset might contain selfies requiring facial analysis, landscapes requiring scene recognition, or screenshots requiring Optical Character Recognition (OCR).
Existing models often struggle to capture these nuances using a single, general-purpose visual encoder. Sentiment is an abstract concept, and social media data is inherently noisy. To address this, researchers have developed VSA-PF (Visual-textual Sentiment Analysis with Pre-trained Features), a holistic framework that leverages specialized "prior knowledge" from models already expert in specific visual domains.
Methodology: The VSA-PF Framework
The VSA-PF architecture employs a four-pronged approach to integrate diverse signals:
- Visual-Textual Branch: This core learnable component uses a Swin Transformer for images and BERTweet for text, fine-tuned directly on sentiment data to capture task-specific nuances.
- Visual Expert Branch: This branch utilizes four fixed, pre-trained "experts" to extract high-level semantic features:
- Facenet: For detecting emotions in portraits.
- YOLOv5: For identifying sentiment-carrying objects (e.g., a birthday cake vs. a car crash).
- DenseNet161: For scene recognition (e.g., a beach vs. a graveyard).
- Tesseract + Sentence-Transformer: For OCR to read text embedded within images.
- CLIP Branch: Leverages OpenAI’s CLIP to model the implicit correspondence between images and text, identifying when a caption and image are semantically aligned.
- Multimodal Fusion: A 3-layer BERT-based fusion network allows these disparate features to "attend" to one another, resolving potential contradictions between visual and textual signals.
Experimental Results and Insights
The model was evaluated on three major benchmarks: MVSA-Single, MVSA-Multiple, and TumEmo (a large-scale dataset with seven fine-grained emotions). VSA-PF achieved new state-of-the-art (SOTA) performance across all three:
- MVSA-Single: Reached 75.58% accuracy, outperforming previous benchmarks like MGNNS.
- TumEmo: Achieved 76.58% accuracy on the complex 7-class task, a significant lead over the previous best of 71.12%.
Key Insights
- The Importance of OCR: A significant portion of social media imagery contains text. Integrating an OCR expert provided a measurable performance boost, highlighting a gap in many previous models.
- Textual Dominance: Ablation studies revealed that the textual branch (BERTweet) remains the most critical component, as adjectives often convey sentiment more directly than visual pixels.
- The Expert Advantage: Leveraging models pre-trained on millions of domain-specific images (like faces or scenes) is more effective than attempting to learn these features from scratch on smaller sentiment datasets.
Conclusion and Future Work
While VSA-PF is highly effective, the utility of specific "experts" varies by dataset. For instance, scene and OCR features were more impactful on Twitter data than on Tumblr data.
Preliminary testing against GPT-4V on the TumEmo dataset showed that the general-purpose model struggled with complex, multi-class sentiment (achieving only 43% accuracy). This suggests that specialized, holistic models like VSA-PF maintain a significant edge in niche tasks. Future research will explore integrating Large Language Models (LLMs) as feature extractors within this framework.
Summary updated