Zero-Shot Visual Sentiment Prediction via Cross-Domain Knowledge Distillation

Yuya Moroto, Yingrui Ye, Keisuke Maeda, Takahiro Ogawa, Ren TogoIEEE Open Journal of Signal Processing 2023 · DOI

Abstract

There are various sentiment theories for categorizing human sentiments into several discrete sentiment categories, which means that the theory used for training sentiment prediction methods does not always match that used in the test phase. As a solution to this problem, zero-shot visual sentiment prediction methods have been proposed to predict unseen sentiments for which no images are available in the training phase. However, the training of these previous zero-shot methods relies on a single sentiment theory, which limits their ability to handle sentiments from other theories. Thus, this article proposes a more robust zero-shot visual sentiment prediction method that can handle cross-domain sentiments defined in different sentiment theories. Specifically, by focusing on the fact that sentiments are abstract concepts common to humans regardless of whether their theories are different, we incorporate knowledge distillation into our method to construct a teacher–student model that can train the implicit relationships between sentiments defined in different sentiment theories. Furthermore, to enhance sentiment discrimination capability and strengthen the implicit relationships between sentiments, we introduce a novel sentiment loss between the teacher and student models. In this way, our model becomes robust to unseen sentiments by exploiting the implicit relationships between sentiments. The contributions of this article are the introduction of knowledge distillation and a novel sentiment loss between the teacher and student models for zero-shot visual sentiment prediction, and improved performance of zero-shot visual sentiment prediction. Experiments on several open datasets demonstrate the effectiveness of the proposed method.

Introduction

Visual Sentiment Prediction (VSP) is the task of training computational models to understand the emotions an image might evoke in a human. A significant challenge in this field is the lack of a unified psychological theory of emotion. Different datasets utilize varying taxonomies, such as Ekman’s six basic emotions, Mikels’ eight categories, or Parrott’s 25 fine-grained sentiments.

Figure 1: The challenge of cross-domain sentiment prediction where training and testing phases utilize different psychological theories.

Most existing models are "theory-locked," struggling when tested on a sentiment framework different from the one used during training. While Zero-Shot Learning (ZSL) aims to predict "unseen" classes, most ZSL-VSP methods still rely on a single sentiment theory. This research addresses this domain gap by proposing a framework that translates emotional knowledge across different taxonomies.

Methods

The authors propose a Cross-domain Knowledge Distillation framework consisting of two main stages: a teacher model and a student model.

Figure 2: The proposed architecture featuring teacher and student model training phases and the shared embedding space.

Teacher-Student Framework

  1. The Teacher Model: Trained on the large-scale T4SA dataset using Ekman’s theory. It learns to map visual features (from ResNet-50) and semantic features (from Adjective-Noun Pairs and tweet text) into a common embedding space where similar emotions are clustered together.
  2. The Student Model: Initialized as a clone of the teacher but trained on a different sentiment theory (e.g., Plutchik’s).
  3. Knowledge Distillation (KD): The student is forced to mimic the teacher’s output distributions, allowing it to inherit the teacher's broad understanding of emotional structures.

Novel Sentiment Loss ($L_{sent}$)

The core innovation is a novel sentiment loss function. It ensures that the relative distances between common sentiments (like "Joy" or "Anger") remain consistent between the teacher's theory and the student's theory. By maintaining these implicit relationships, the model can generalize to entirely new, "unseen" labels.

Experimental Results

The proposed method was evaluated on several diverse datasets, including real-world social media images (FI), professional photography (ART), and abstract paintings (ABST).

Key findings include:

  • FI Dataset: The model achieved 70.72% accuracy for two unseen sentiments, significantly outperforming previous state-of-the-art methods.
  • Abstract Art (ABST): In a test of 8 sentiments, the model reached 22.14% accuracy. While lower than social media benchmarks, this is a strong result given the human baseline for this subjective task is 38.61%.
  • Fine-grained Prediction: On the WEBEmo dataset, the model showed superior performance in complex categories like "Love" (39.17%) and "Anger" (31.41%).

Conclusion

By treating sentiments as abstract constants and focusing on their inter-relationships, this method bridges the gap between disparate psychological theories. The results demonstrate that knowledge distillation is a powerful tool for creating robust, theory-agnostic sentiment prediction models. Future work will focus on automated hyperparameter tuning to further optimize the balance between visual, distillation, and sentiment losses.

Summary updated

Abstract

There are various sentiment theories for categorizing human sentiments into several discrete sentiment categories, which means that the theory used for training sentiment prediction methods does not always match that used in the test phase. As a solution to this problem, zero-shot visual sentiment prediction methods have been proposed to predict unseen sentiments for which no images are available in the training phase. However, the training of these previous zero-shot methods relies on a single sentiment theory, which limits their ability to handle sentiments from other theories. Thus, this article proposes a more robust zero-shot visual sentiment prediction method that can handle cross-domain sentiments defined in different sentiment theories. Specifically, by focusing on the fact that sentiments are abstract concepts common to humans regardless of whether their theories are different, we incorporate knowledge distillation into our method to construct a teacher–student model that can train the implicit relationships between sentiments defined in different sentiment theories. Furthermore, to enhance sentiment discrimination capability and strengthen the implicit relationships between sentiments, we introduce a novel sentiment loss between the teacher and student models. In this way, our model becomes robust to unseen sentiments by exploiting the implicit relationships between sentiments. The contributions of this article are the introduction of knowledge distillation and a novel sentiment loss between the teacher and student models for zero-shot visual sentiment prediction, and improved performance of zero-shot visual sentiment prediction. Experiments on several open datasets demonstrate the effectiveness of the proposed method.
BibTeX
@article{moroto2023zero,
  author = {Yuya Moroto and Yingrui Ye and Keisuke Maeda and Takahiro Ogawa and Ren Togo},
  title = {{Zero-Shot Visual Sentiment Prediction via Cross-Domain Knowledge Distillation}},
  journal = {IEEE Open Journal of Signal Processing},
  year = {2023},
  doi = {10.1109/ojsp.2023.3344079},
}
Figure