Inferring Sentiment from Web Images with Joint Inference on Visual and Social Cues: A Regulated Matrix Factorization Approach

Yilin Wang, Yuheng Hu, Subbarao Kambhampati, Baoxin Li

Abstract

In this paper, we study the problem of understanding human sentiments from large scale collection of Internet images based on both image features and contextual social network information (such as friend comments and user description). Despite the great strides in analyzing user sentiment based on text information, the analysis of sentiment behind the image content has largely been ignored. Thus, we extend the significant advances in text-based sentiment prediction tasks to the higher-level challenge of predicting the underlying sentiments behind the images. We show that neither visual features nor the textual features are by themselves sufficient for accurate sentiment labeling. Thus, we provide a way of using both of them. We leverage the low-level visual features and mid-level attributes of an image, and formulate sentiment prediction problem as a non-negative matrix tri-factorization framework, which has the flexibility to incorporate multiple modalities of information and the capability to learn from heterogeneous features jointly. We develop an optimization algorithm for finding a local-optima solution under the proposed framework. With experiments on two large-scale datasets, we show that the proposed method improves significantly over existing state-of-the-art methods.

The Visual Affective Gap

While sentiment analysis for text is a mature field, interpreting the emotional weight of images remains a significant challenge. This difficulty stems from the "visual affective gap": the disconnect between low-level pixel data and high-level human emotion.

A visual example of the affective gap: crying can represent joy or sadness depending on context.

For instance, a standard computer vision algorithm might detect a "crying face." Without context, it cannot distinguish between tears of joy at a wedding or proposal and tears of sadness over a lost toy. Purely visual models struggle with this ambiguity, while purely textual models often fail because social media captions are frequently sparse, sarcastic, or missing entirely. This research proposes a method to fuse these heterogeneous data sources for a more accurate emotional "read."

The RSAI Framework

The authors introduce RSAI (Robust Sentiment Analysis for Images), a framework built on Non-negative Matrix Tri-Factorization (NMTF). This approach bridges the gap by jointly modeling visual features and social context.

The RSAI framework jointly processes visual features and social text to predict sentiment.

Feature Extraction and Modeling

The framework operates by extracting two primary types of features:

  1. Visual Features: The model uses low-level color histograms and mid-level attributes. A key innovation is the use of Adjective Noun Pairs (ANPs)—such as "misty forest" or "happy baby"—to map visual objects to specific sentimental meanings.
  2. Social/Textual Features: Terms are extracted from user descriptions and friend comments.

The problem is formulated as a matrix factorization task where an input image-feature matrix is decomposed into three factors: image-term, term-sentiment, and sentiment-features.

Regulated Optimization

Unlike standard factorization, RSAI is "regulated" by three types of prior knowledge:

  • Sentiment Lexicons: Using the MPQA corpus for baseline word sentiment.
  • Visual Sentiment Ontology: Utilizing pre-defined sentiment strengths for specific ANPs.
  • Partial Labels: Incorporating known sentiment labels for a subset of images to guide semi-supervised learning.

Experimental Results

The study utilized two large-scale real-world datasets: Flickr (120,221 images) and Instagram (130,230 images). Ground truth was established via manual labeling of 20,000 Flickr images by human subjects.

RSAI consistently outperformed state-of-the-art benchmarks like Senti API (text-based), SentiBank (visual-based), and EL (graphical model). On Flickr, it achieved an accuracy of 0.57 (vs. 0.48 baseline). On Instagram, the performance reached 0.62 accuracy (vs. 0.54 baseline).

Key Insights

  • The Power of Multi-Modality: Neither visual nor textual features are sufficient alone. The "visual affective gap" is bridged when pixels are interpreted through the lens of surrounding social conversation.
  • ANPs as a Semantic Bridge: Adjective Noun Pairs provide the "what" (noun) and the "how" (adjective), making visual data machine-readable for emotional tasks.
  • Robustness to Sparse Data: The matrix factorization approach allows the model to make reasonable inferences even when social text is sparse by relying on learned visual-sentiment factors.

Conclusion and Future Work

While RSAI shows significant promise, challenges remain regarding noisy social media language and computational complexity for real-time scaling. The authors suggest that future iterations could integrate deep neural networks to replace manual feature extraction, further closing the affective gap between pixels and human sentiment.

Summary updated

Abstract

In this paper, we study the problem of understanding human sentiments from large scale collection of Internet images based on both image features and contextual social network information (such as friend comments and user description). Despite the great strides in analyzing user sentiment based on text information, the analysis of sentiment behind the image content has largely been ignored. Thus, we extend the significant advances in text-based sentiment prediction tasks to the higher-level challenge of predicting the underlying sentiments behind the images. We show that neither visual features nor the textual features are by themselves sufficient for accurate sentiment labeling. Thus, we provide a way of using both of them. We leverage the low-level visual features and mid-level attributes of an image, and formulate sentiment prediction problem as a non-negative matrix tri-factorization framework, which has the flexibility to incorporate multiple modalities of information and the capability to learn from heterogeneous features jointly. We develop an optimization algorithm for finding a local-optima solution under the proposed framework. With experiments on two large-scale datasets, we show that the proposed method improves significantly over existing state-of-the-art methods.
BibTeX
@inproceedings{wang2021inferring,
  author = {Yilin Wang and Yuheng Hu and Subbarao Kambhampati and Baoxin Li},
  title = {{Inferring Sentiment from Web Images with Joint Inference on Visual and Social Cues: A Regulated Matrix Factorization Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/icwsm.v9i1.14608},
}
Figure