Abstract
Visual-textual sentiment analysis aims to predict sentiment from image-text pairs. The primary challenge lies in learning effective visual features from diverse social media content. We propose VSA-PF, a framework that integrates powerful expert visual features—specializing in faces, scenes, objects, and OCR—with a CLIP-based alignment branch and a transformer-based fusion network. Experiments on MVSA and TumEmo datasets demonstrate that our method achieves state-of-the-art performance by leveraging specialized prior knowledge.
Introduction: The Complexity of Social Media Sentiment
Sentiment analysis on social media extends beyond simple text processing. Users often express emotions through complex combinations of images and text, such as memes, screenshots, or photos where sentiment is implied by context. For instance, a photo of a "WAIT FOR GAP" sign might appear neutral, but when paired with a complaint about pedestrian rights, it conveys a negative sentiment.
Existing models often struggle with these abstract and multi-faceted expressions. A general-purpose neural network may fail to capture nuances like text embedded within images or subtle facial expressions. To address this, we introduce VSA-PF (Visual-textual Sentiment Analysis with Pre-trained Features), which utilizes a "panel of experts" to provide high-level semantic priors.
Methodology
The VSA-PF framework consists of four primary branches fused by a central transformer-based "brain":
- Visual-Textual Branch: Employs a Swin Transformer for images and BERTweet for text, fine-tuned directly on sentiment data to capture domain-specific patterns.
- Visual Expert Branch: Utilizes four fixed, pre-trained models to extract specialized features:
- Face Recognition (MTCNN/Facenet): Encodes facial expressions.
- Object Detection (YOLOv5): Identifies and counts relevant objects.
- Scene Recognition (DenseNet161): Categorizes the environment (e.g., beach vs. hospital).
- OCR (Tesseract/Sentence-Transformer): Extracts and encodes text found within images.
- CLIP Branch: Uses OpenAI’s CLIP model to model the implicit correspondence between the image and text modalities.
- Multimodal Fusion Network: A 3-layer BERT-based transformer encoder that fuses features from all branches to make the final sentiment prediction.
Experimental Evaluation
The model was evaluated on three benchmark datasets: MVSA-Single, MVSA-Multiple, and TumEmo (a large-scale Tumblr dataset with seven fine-grained emotion categories).
Key Results
VSA-PF established new state-of-the-art (SOTA) benchmarks across all tested datasets:
- TumEmo: Achieved 76.58% accuracy, significantly outperforming previous models like CLMLF (71.12%).
- MVSA-Single: Reached 75.58% accuracy.
- Robustness: The model proved particularly effective at handling "weakly sentiment" images—visuals that are neutral in isolation but gain emotional weight when paired with specific text.
Discussion and Insights
Ablation studies revealed several critical insights into multimodal sentiment analysis:
- The Critical Role of OCR: Sentiment in social media is frequently embedded within the image itself. Removing the OCR component led to a significant performance drop, highlighting its necessity for understanding memes and quotes.
- Text Dominance: Despite the visual complexity, the textual encoder (BERTweet) remains the strongest individual predictor, as linguistic cues like adjectives are more direct indicators of sentiment.
- Expert Priors: Specialized models for faces and scenes allow the central system to focus on interpreting sentiment rather than basic object identification, effectively leveraging millions of images worth of prior knowledge.
Conclusion and Future Directions
While VSA-PF advances the state-of-the-art, challenges remain, particularly in processing handwritten text which often baffles current OCR engines. Future work will explore the integration of Large Language Models (LLMs) and Large Visual Models (LVMs). Preliminary tests suggest that while general models like GPT-4V are powerful, specialized frameworks like VSA-PF still maintain an edge in these complex, domain-specific sentiment tasks.
Summary updated