Predicting Sentiments in Image Advertisements using Semantic Relations among Sentiment Labels

Stephen Pilli, Manasi Patwardhan, Niranjan Pedanekar, Shirish KarandeWorkshops @ CVPR 2020 · DOI

Introduction

In the digital advertising landscape, identifying the sentiment an image evokes is as critical as identifying its subject matter. However, while computer vision has excelled at object detection, sentiment analysis in advertisements remains a significant challenge. Unlike natural images, advertisements utilize highly stylized visual rhetoric and symbolic language to evoke complex, often conflicting emotions. This paper addresses the limitations of traditional models by introducing a neural network architecture that leverages Graph Convolutional Networks (GCNs) to model the semantic relationships between sentiment labels.

The Challenge of Emotional Ambivalence and Label Independence

Traditional machine learning approaches often treat sentiment labels as independent categories. This "siloed" approach fails to account for two core realities of advertising:

  1. Emotional Ambivalence: Advertisements frequently aim to provoke multiple, sometimes contradictory, emotions simultaneously (e.g., "shock" and "calm") to capture attention and drive engagement.
  2. Semantic Relatedness: Emotions are not isolated; "calm" and "peaceful" share significant semantic overlap, whereas "shocked" is distinct. Standard models do not inherently understand these nuances.

To bridge this gap, the researchers utilized a hierarchy of labels, categorizing sentiments into "representative" groups and their semantically related sub-labels.

Examples of representative sentiments and their semantically related sub-labels used to model relationships between emotions.

Methodology: Joint Semantic Learning

The proposed architecture employs a multi-stage process to translate pixels into complex emotional insights:

  • Visual Feature Extraction: A pre-trained ResNet-152 model extracts high-level visual features from the advertisement images.
  • Semantic Embedding: These features are projected into a sentiment label space initialized with word2vec embeddings. This provides the model with a baseline understanding of linguistic relationships between sentiment terms.
  • Graph Convolutional Network (GCN): The GCN acts as the core reasoning engine, learning an adjacency matrix that captures pairwise similarities between 115 different sentiment labels. This allows the prediction of one label (e.g., "cheerful") to inform and refine the prediction of related labels (e.g., "happy").
  • The Max-Norm Loss Function: Unsupervised cluster analysis revealed that sentiment labels do not follow a standard Euclidean distribution. The researchers implemented a modified hinge rank loss function based on the max-norm (infinity norm), which better represents the semantic clusters in high-dimensional space.

Dataset Analysis and the "Creative" Trap

The study utilized a large-scale dataset of 64,140 advertisement images, with 30,340 specifically annotated for sentiments by human raters. A key discovery during the analysis was the significant skew toward the label "creative."

Distribution of sentiment labels across the advertisement dataset, highlighting the significant skew toward the 'creative' label.

As shown in the distribution above, "creative" appears as a dominant outlier. The authors identified this as "label noise," noting that human annotators often marked the advertisement itself as creative rather than describing the sentiment it evoked. By undersampling this noisy label, the model's performance on more descriptive, emotion-focused sentiments improved significantly.

Key Results

The proposed model demonstrated a substantial improvement over the previous state-of-the-art "M&M" (Multi-modal Multi-task) model:

  • Mean Average Precision (mAP): Achieved a 14% relative increase (from 0.292 to 0.332).
  • GCN Impact: When compared to a standard GCN without semantic enhancements, the proposed model showed a 49% improvement in mAP, validating the necessity of visual context in graph-based label reasoning.
  • Zero-Shot Potential: By leveraging word embeddings, the model demonstrated an ability to make informed predictions for sentiments with sparse training data, relying on its learned semantic graph.

Conclusion

By moving away from label independence and embracing the semantic geometry of human emotion, this research provides a more robust framework for understanding the visual rhetoric of advertising. Future work will focus on refining datasets to eliminate annotation biases and further exploring zero-shot learning to predict entirely new sentiment categories based solely on semantic relationships.

Summary updated

BibTeX
@inproceedings{pilli2020predicting,
  author = {Stephen Pilli and Manasi Patwardhan and Niranjan Pedanekar and Shirish Karande},
  title = {{Predicting Sentiments in Image Advertisements using Semantic Relations among Sentiment Labels}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00212},
}
Figure