Probing Sentiment-Oriented PreTraining Inspired by Human Sentiment Perception Mechanism

Tinglei Feng, Jiaxuan Liu, Jufeng YangCVPR 2023 · OpenReview

Abstract

Pre-training of deep convolutional neural networks (DC-NNs) plays a crucial role in the field of visual sentiment analysis (VSA). Most proposed methods employ the off-the-shelf backbones pre-trained on large-scale object classification datasets (i.e., ImageNet). While it boosts performance for a big margin against initializing model states from random, we argue that DCNNs simply pre-trained on ImageNet may excessively focus on recognizing objects, but failed to provide high-level concepts in terms of sentiment. To address this long-term overlooked problem, we propose a sentiment-oriented pre-training method that is built upon human visual sentiment perception (VSP) mechanism. Specifically, we factorize the process of VSP into three steps, namely stimuli taking, holistic organizing, and high-level perceiving. From imitating each VSP step, a total of three models are separately pre-trained via our devised sentiment-aware tasks that contribute to excavating sentiment-discriminated representations. Moreover, along with our elaborated multi-model amalgamation strategy, the prior knowledge learned from each perception step can be effectively transferred into a single target model, yielding substantial performance gains. Finally, we verify the superiorities of our proposed method over extensive experiments, covering mainstream VSA tasks from single-label learning (SLL), multi-label learning (MLL), to label distribution learning (LDL). Experiment results demonstrate that our proposed method leads to unanimous improvements in these downstream tasks. Our code is released on https://github.com/tinglyfeng/sentiment_pretraining.

Introduction: The Affective Gap in Computer Vision

In the realm of computer vision, the standard paradigm has long been to pre-train models on ImageNet before fine-tuning them for specific downstream tasks. While this approach is highly effective for object recognition, it often falters in Visual Sentiment Analysis (VSA). The core issue lies in the "affective gap"—the disconnect between low-level visual features and high-level emotional responses.

Standard object-centric pre-training teaches models to distinguish between physical entities, such as different breeds of dogs. However, sentiment is inherently subjective and context-dependent. A model trained solely on objects may fail to distinguish the emotional difference between a "happy dog" and a "vicious dog," as the underlying object remains the same. To bridge this gap, this research proposes a pre-training framework inspired by the human Visual Sentiment Perception (VSP) mechanism, shifting the focus from object identification to sentiment-aware representation learning.

The Human-Inspired Framework

The researchers decompose the human VSP process into three chronological stages, designing specific pre-training tasks for each to capture different layers of sentiment-relevant information.

Overview of the three-stage sentiment-oriented pre-training framework.

1. Stimuli Taking (ST)

This initial stage mimics the retina's reception of light, focusing on fundamental visual elements like color and texture. The pre-training tasks include Colorization (reconstructing color from grayscale) and Super-Resolution (recovering fine-grained textures). These tasks force the model to attend to the low-level stimuli that often trigger immediate emotional responses.

2. Holistic Organizing (HO)

Mirroring the primary visual cortex's role in building a global scene map, this stage focuses on spatial relationships and environmental context. Tasks include Scene Recognition and solving Jigsaw Puzzles. By understanding how elements are organized within a frame, the model learns to perceive the "atmosphere" of a scene.

3. High-level Perceiving (HP)

This final stage represents the brain's higher-order functions in identifying objects and their interactions. The researchers utilize Adjective-Noun Pair (ANP) Prediction and Image Captioning (using the ArtEmis dataset). By focusing on adjectives (e.g., "misty," "abandoned"), the model learns that the state of an object is more critical for sentiment than the object itself.

Multi-Model Amalgamation

Training a single model on all six tasks simultaneously can lead to optimization conflicts, as different tasks may pull the model's weights in opposing directions. To solve this, the authors employ a Multi-Model Amalgamation strategy. Three specialized models are first pre-trained for the ST, HO, and HP stages. Their collective knowledge is then transferred into a single target model through feature-level and logit-level regularization. This allows the target model to benefit from the specialized expertise of each stage without the interference of multi-task training.

Experimental Results

The proposed method was evaluated across a wide range of VSA tasks, including single-label learning (SLL), multi-label learning (MLL), and label distribution learning (LDL). The results consistently showed that sentiment-oriented pre-training outperforms standard ImageNet initialization.

Performance comparison across different backbones and training strategies.

On the Flickr and Instagram (FI) dataset, a ResNet50 backbone using this method achieved an accuracy of 70.7%, a significant improvement over the 67.0% achieved with ImageNet pre-training. Furthermore, the method demonstrated superior performance compared to state-of-the-art self-supervised learning (SSL) methods like SimCLR and MoCo v3. The validation loss curves also indicate that the proposed method leads to faster and more stable convergence compared to traditional approaches.

Conclusion

By aligning AI pre-training with the human mechanism of sentiment perception, this work addresses the long-standing affective gap in VSA. While the multi-model amalgamation process introduces additional computational overhead during the pre-training phase, the resulting performance gains across diverse sentiment tasks justify the investment. This research highlights that for AI to truly understand how an image "feels," it must look beyond the objects and perceive the world through a more human-centric lens.

Summary updated

Abstract

Pre-training of deep convolutional neural networks (DC-NNs) plays a crucial role in the field of visual sentiment analysis (VSA). Most proposed methods employ the off-the-shelf backbones pre-trained on large-scale object classification datasets (i.e., ImageNet). While it boosts performance for a big margin against initializing model states from random, we argue that DCNNs simply pre-trained on ImageNet may excessively focus on recognizing objects, but failed to provide high-level concepts in terms of sentiment. To address this long-term overlooked problem, we propose a sentiment-oriented pre-training method that is built upon human visual sentiment perception (VSP) mechanism. Specifically, we factorize the process of VSP into three steps, namely stimuli taking, holistic organizing, and high-level perceiving. From imitating each VSP step, a total of three models are separately pre-trained via our devised sentiment-aware tasks that contribute to excavating sentiment-discriminated representations. Moreover, along with our elaborated multi-model amalgamation strategy, the prior knowledge learned from each perception step can be effectively transferred into a single target model, yielding substantial performance gains. Finally, we verify the superiorities of our proposed method over extensive experiments, covering mainstream VSA tasks from single-label learning (SLL), multi-label learning (MLL), to label distribution learning (LDL). Experiment results demonstrate that our proposed method leads to unanimous improvements in these downstream tasks. Our code is released on https://github.com/tinglyfeng/sentiment_pretraining.
BibTeX
@inproceedings{feng2023probing,
  author = {Tinglei Feng and Jiaxuan Liu and Jufeng Yang},
  title = {{Probing Sentiment-Oriented PreTraining Inspired by Human Sentiment Perception Mechanism}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=mQ2YQjN4Qq},
}
Figure