Sentiment Analysis of Fashion Related Posts in Social Media

Yifei Yuan, Wai Lam

Abstract

The role of social media in fashion industry has been blooming as the years have continued on. In this work, we investigate sentiment analysis for fashion related posts in social media platforms. There are two main challenges of this task. On the first place, information of different modalities must be jointly considered to make the final predictions. On the second place, some unique fashion related attributes should be taken into account. While most existing works focus on traditional multimodal sentiment analysis, they always fail to exploit the fashion related attributes in this task. We propose a novel framework that jointly leverages the image vision, post text, as well as fashion attribute modality to determine the sentiment category. One characteristic of our model is that it extracts fashion attributes and integrates them with the image vision information for effective representation. Furthermore, it exploits the mutual relationship between the fashion attributes and the post texts via a mutual attention mechanism. Since there is no existing dataset suitable for this task, we prepare a large-scale sentiment analysis dataset of over 12k fashion related social media posts. Extensive experiments are conducted to demonstrate the effectiveness of our model.

Introduction

Social media has become a primary arena for fashion discourse, yet automated sentiment analysis in this domain remains challenging. Traditional models often struggle with the complex interplay between visual cues and textual nuances like sarcasm or domain-specific slang. For instance, a phrase like "this pattern kills it" requires context to determine if it is a compliment or a critique. This research addresses these challenges by proposing a novel framework that treats fashion attributes as a distinct modality and introducing a large-scale benchmark dataset of over 12,000 Instagram posts.

Methodology

The proposed framework utilizes a tripartite architecture to process fashion-related social media posts, integrating vision, text, and specific fashion metadata.

The proposed multimodal framework for fashion sentiment analysis, integrating vision, text, and fashion attributes.

The architecture consists of three specialized modules:

  1. Fashion-Aware Vision Composition (VA Module): This module extracts discrete attributes (e.g., "sleeveless," "chiffon") using a tagging tool and fuses them with raw visual features from a ResNet backbone using a Compositional Autoencoder (ComposeAE).
  2. Fashion-Aware Text Composition (TA Module): This module handles the relationship between the post's caption and the fashion items. It employs a mutual attention mechanism to learn which parts of the text are relevant to specific fashion attributes.
  3. Vision Text Composition (VT Module): A multimodal Transformer integrates image features—specifically focusing on detected faces and fashion item bounding boxes—with text features to learn a dense joint representation.

The visual processing component is particularly noteworthy for its focus on specific regions of interest. By identifying both the wearer's facial expression and the specific fashion items, the model can better distinguish between personal sentiment and promotional content.

Example of region detection, identifying faces and fashion items for focused visual analysis.

Datasets

To evaluate the model, the authors curated the Instagram Fashion Dataset, consisting of 12,061 manually labeled posts. Hashtags such as #fashion and #ootd were used to gather positive/neutral samples, while #uglyclothes and #fashionyoudontunderstand targeted negative sentiments. Each entry includes an image, text, and a sentiment label (Positive, Neutral, or Negative).

Experimental Results

The proposed model achieved an accuracy of 67.58% and an F1 score of 67.56% on the Instagram dataset, significantly outperforming unimodal baselines (Text-only: 57.96%; Image-only: 47.04%). Ablation studies confirmed that all three modules (VA, TA, and VT) are critical for performance.

A key finding was the role of fashion attributes as a "semantic bridge" between high-level text and low-level pixels. While text-only models are strong, the addition of visual context and attributes is essential for handling ambiguous "anti-fashion" posts where the text might be sarcastic. The model also highlighted the difficulty of classifying "Neutral" posts, which are often professional advertisements that mimic positive visual cues but lack genuine sentiment.

Conclusion

While the framework demonstrates state-of-the-art performance, challenges remain in handling deep sarcasm and the temporal nature of fashion trends. Future work will explore incorporating "trendiness" scores and more sophisticated linguistic models to better decode complex social media expressions.

Summary updated

Abstract

The role of social media in fashion industry has been blooming as the years have continued on. In this work, we investigate sentiment analysis for fashion related posts in social media platforms. There are two main challenges of this task. On the first place, information of different modalities must be jointly considered to make the final predictions. On the second place, some unique fashion related attributes should be taken into account. While most existing works focus on traditional multimodal sentiment analysis, they always fail to exploit the fashion related attributes in this task. We propose a novel framework that jointly leverages the image vision, post text, as well as fashion attribute modality to determine the sentiment category. One characteristic of our model is that it extracts fashion attributes and integrates them with the image vision information for effective representation. Furthermore, it exploits the mutual relationship between the fashion attributes and the post texts via a mutual attention mechanism. Since there is no existing dataset suitable for this task, we prepare a large-scale sentiment analysis dataset of over 12k fashion related social media posts. Extensive experiments are conducted to demonstrate the effectiveness of our model.
BibTeX
@inproceedings{yuan2021sentiment,
  author = {Yifei Yuan and Wai Lam},
  title = {{Sentiment Analysis of Fashion Related Posts in Social Media}},
  booktitle = {WSDM},
  year = {2021},
  doi = {10.1145/3488560.3498423},
}
Figure