Improving Visual-textual Sentiment Analysis by Fusing Expert Features

Junyu Chen, Jie An, Hanjia Lyu, Jiebo Luo

Abstract

Visual-textual sentiment analysis aims to predict sentiment from image-text pairs. The primary challenge lies in learning effective visual features from diverse social media content. We propose VSA-PF, a framework that integrates powerful expert visual features—specializing in faces, scenes, objects, and OCR—with a CLIP-based alignment branch and a transformer-based fusion network. Experiments on MVSA and TumEmo datasets demonstrate that our method achieves state-of-the-art performance by leveraging specialized prior knowledge.

Introduction: The Complexity of Social Media Sentiment

Sentiment analysis on social media extends beyond simple text processing. Users often express emotions through complex combinations of images and text, such as memes, screenshots, or photos where sentiment is implied by context. For instance, a photo of a "WAIT FOR GAP" sign might appear neutral, but when paired with a complaint about pedestrian rights, it conveys a negative sentiment.

Examples of image-text pairs and their associated sentiment labels from the study datasets.

Existing models often struggle with these abstract and multi-faceted expressions. A general-purpose neural network may fail to capture nuances like text embedded within images or subtle facial expressions. To address this, we introduce VSA-PF (Visual-textual Sentiment Analysis with Pre-trained Features), which utilizes a "panel of experts" to provide high-level semantic priors.

Methodology

The VSA-PF framework consists of four primary branches fused by a central transformer-based "brain":

  1. Visual-Textual Branch: Employs a Swin Transformer for images and BERTweet for text, fine-tuned directly on sentiment data to capture domain-specific patterns.
  2. Visual Expert Branch: Utilizes four fixed, pre-trained models to extract specialized features:
  3. Face Recognition (MTCNN/Facenet): Encodes facial expressions.
  4. Object Detection (YOLOv5): Identifies and counts relevant objects.
  5. Scene Recognition (DenseNet161): Categorizes the environment (e.g., beach vs. hospital).
  6. OCR (Tesseract/Sentence-Transformer): Extracts and encodes text found within images.
  7. CLIP Branch: Uses OpenAI’s CLIP model to model the implicit correspondence between the image and text modalities.
  8. Multimodal Fusion Network: A 3-layer BERT-based transformer encoder that fuses features from all branches to make the final sentiment prediction.

Experimental Evaluation

The model was evaluated on three benchmark datasets: MVSA-Single, MVSA-Multiple, and TumEmo (a large-scale Tumblr dataset with seven fine-grained emotion categories).

Key Results

VSA-PF established new state-of-the-art (SOTA) benchmarks across all tested datasets:

  • TumEmo: Achieved 76.58% accuracy, significantly outperforming previous models like CLMLF (71.12%).
  • MVSA-Single: Reached 75.58% accuracy.
  • Robustness: The model proved particularly effective at handling "weakly sentiment" images—visuals that are neutral in isolation but gain emotional weight when paired with specific text.

Discussion and Insights

Ablation studies revealed several critical insights into multimodal sentiment analysis:

  • The Critical Role of OCR: Sentiment in social media is frequently embedded within the image itself. Removing the OCR component led to a significant performance drop, highlighting its necessity for understanding memes and quotes.
  • Text Dominance: Despite the visual complexity, the textual encoder (BERTweet) remains the strongest individual predictor, as linguistic cues like adjectives are more direct indicators of sentiment.
  • Expert Priors: Specialized models for faces and scenes allow the central system to focus on interpreting sentiment rather than basic object identification, effectively leveraging millions of images worth of prior knowledge.

Conclusion and Future Directions

While VSA-PF advances the state-of-the-art, challenges remain, particularly in processing handwritten text which often baffles current OCR engines. Future work will explore the integration of Large Language Models (LLMs) and Large Visual Models (LVMs). Preliminary tests suggest that while general models like GPT-4V are powerful, specialized frameworks like VSA-PF still maintain an edge in these complex, domain-specific sentiment tasks.

Summary updated

Abstract

Visual-textual sentiment analysis aims to predict sentiment with the input of a pair of image and text. The main challenge of visual-textual sentiment analysis is how to learn effective visual features for sentiment prediction since input images are often very diverse. To address this challenge, we propose a new method that improves visual-textual sentiment analysis by introducing powerful expert visual features. The proposed method consists of four parts: (1) a visual-textual branch to learn features directly from data for sentiment analysis, (2) a visual expert branch with a set of pre-trained "expert" encoders to extract effective visual features, (3) a CLIP branch to implicitly model visual-textual correspondence, and (4) a multimodal feature fusion network based on either BERT or MLP to fuse multimodal features and make sentiment prediction. Extensive experiments on three datasets show that our method produces better visual-textual sentiment analysis performance than existing methods.
BibTeX
@inproceedings{chen2022improving,
  author = {Junyu Chen and Jie An and Hanjia Lyu and Jiebo Luo},
  title = {{Improving Visual-textual Sentiment Analysis by Fusing Expert Features}},
  booktitle = {MIPR},
  year = {2022},
}
Figure