Automatic Fashion Knowledge Extraction from Social Media

Yunshan Ma, Lizi Liao, Tat-Seng Chua

Abstract

Fashion knowledge plays a pivotal role in helping people in their dressing. In this paper, we present a novel system to automatically harvest fashion knowledge from social media. It unifies three tasks of occasion, person and clothing discovery from multiple modalities of images, texts and metadata. A contextualized fashion concept learning model is applied to leverage the rich contextual information for improving the fashion concept learning performance. At the same time, to counter the label noise within training data, we employ a weak label modeling method to further boost the performance. We build a website to demonstrate the quality of fashion knowledge extracted by our system.

Introduction

While computer vision has made significant strides in identifying specific garments, such as a "red dress," fashion is fundamentally a social and contextual phenomenon. The meaning and appropriateness of an outfit are defined by the interplay between the wearer and the setting. Current fashion AI often lacks this context, focusing narrowly on item recognition while ignoring the social media noise—irrelevant hashtags and lack of expert labels—that characterizes real-world data.

To bridge this gap, we propose a system that extracts structured "fashion knowledge" by linking three key pillars: occasion, person, and clothing. By treating fashion as a relationship rather than a single label, we can better understand how people actually dress in various social contexts.

The system extracts a structured triplet of <occasion, person, clothes> from multi-modal social media posts.

Methodology

Our framework processes raw social media data through a multi-layered pipeline. It begins with a crawler that gathers Instagram posts, which are then filtered to ensure they contain clear, high-quality images of people. This involves specific face-to-body and body-to-image height ratios to maintain data quality.

Contextualized Fashion Concept Learning

The core of our system is a multi-modal learning model. We utilize Image and Text CNNs to extract features from both the visual and textual components of a post. These features are integrated using Bi-directional LSTMs (Bi-LSTMs) to model the correlations between different tasks. For instance, the model can learn that certain clothing attributes, like a "collar," are frequently associated with "conference" settings.

The contextualized fashion concept learning architecture, integrating image and text features with weak label modeling.

Weak Label Modeling

Social media data is inherently "noisy." Users often apply hashtags that are only tangentially related to the image content. To address this, we implemented a weak label modeling approach. This includes a label correction module that uses a learned transition matrix to distinguish between noisy user-provided tags and the true underlying fashion concepts, significantly boosting the model's robustness.

Key Insights and Results

The most significant contribution of this work is the conceptualization of fashion knowledge as a triplet: <occasion, person, clothes>. This structure allows for the creation of a searchable knowledge graph. Our system enables complex queries, such as searching for "females wearing dresses to a prom," which is far more useful than simple keyword-based image search.

Experimental results demonstrate that modeling the dependencies between occasions and clothing attributes via Bi-LSTMs yields superior performance compared to treating these tasks independently. The system successfully handles a taxonomy of 10 occasions, 21 clothing categories, and 50 attribute values, providing a comprehensive view of real-world fashion trends.

Conclusion and Future Work

We have demonstrated an end-to-end system for automatic fashion knowledge extraction that thrives on the rich, albeit noisy, environment of social media. Future work will focus on temporal dynamics, specifically "trending analysis" to track how these fashion triplets evolve over time. Additionally, we aim to explore influence diffusion to understand how trends originate and spread across social networks.

Summary updated

Abstract

Fashion knowledge plays a pivotal role in helping people in their dressing. In this paper, we present a novel system to automatically harvest fashion knowledge from social media. It unifies three tasks of occasion, person and clothing discovery from multiple modalities of images, texts and metadata. A contextualized fashion concept learning model is applied to leverage the rich contextual information for improving the fashion concept learning performance. At the same time, to counter the label noise within training data, we employ a weak label modeling method to further boost the performance. We build a website to demonstrate the quality of fashion knowledge extracted by our system.
BibTeX
@inproceedings{ma2019automatic,
  author = {Yunshan Ma and Lizi Liao and Tat-Seng Chua},
  title = {{Automatic Fashion Knowledge Extraction from Social Media}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=iDyKAM5uR4},
}
Figure