Abstract
Fashion knowledge plays a pivotal role in helping people in their dressing. In this paper, we present a novel system to automatically harvest fashion knowledge from social media. It unifies three tasks of occasion, person and clothing discovery from multiple modalities of images, texts and metadata. A contextualized fashion concept learning model is applied to leverage the rich contextual information for improving the fashion concept learning performance. At the same time, to counter the label noise within training data, we employ a weak label modeling method to further boost the performance. We build a website to demonstrate the quality of fashion knowledge extracted by our system.
Introduction
While computer vision has made significant strides in identifying specific garments, such as a "red dress," fashion is fundamentally a social and contextual phenomenon. The meaning and appropriateness of an outfit are defined by the interplay between the wearer and the setting. Current fashion AI often lacks this context, focusing narrowly on item recognition while ignoring the social media noise—irrelevant hashtags and lack of expert labels—that characterizes real-world data.
To bridge this gap, we propose a system that extracts structured "fashion knowledge" by linking three key pillars: occasion, person, and clothing. By treating fashion as a relationship rather than a single label, we can better understand how people actually dress in various social contexts.
Methodology
Our framework processes raw social media data through a multi-layered pipeline. It begins with a crawler that gathers Instagram posts, which are then filtered to ensure they contain clear, high-quality images of people. This involves specific face-to-body and body-to-image height ratios to maintain data quality.
Contextualized Fashion Concept Learning
The core of our system is a multi-modal learning model. We utilize Image and Text CNNs to extract features from both the visual and textual components of a post. These features are integrated using Bi-directional LSTMs (Bi-LSTMs) to model the correlations between different tasks. For instance, the model can learn that certain clothing attributes, like a "collar," are frequently associated with "conference" settings.
Weak Label Modeling
Social media data is inherently "noisy." Users often apply hashtags that are only tangentially related to the image content. To address this, we implemented a weak label modeling approach. This includes a label correction module that uses a learned transition matrix to distinguish between noisy user-provided tags and the true underlying fashion concepts, significantly boosting the model's robustness.
Key Insights and Results
The most significant contribution of this work is the conceptualization of fashion knowledge as a triplet: <occasion, person, clothes>. This structure allows for the creation of a searchable knowledge graph. Our system enables complex queries, such as searching for "females wearing dresses to a prom," which is far more useful than simple keyword-based image search.
Experimental results demonstrate that modeling the dependencies between occasions and clothing attributes via Bi-LSTMs yields superior performance compared to treating these tasks independently. The system successfully handles a taxonomy of 10 occasions, 21 clothing categories, and 50 attribute values, providing a comprehensive view of real-world fashion trends.
Conclusion and Future Work
We have demonstrated an end-to-end system for automatic fashion knowledge extraction that thrives on the rich, albeit noisy, environment of social media. Future work will focus on temporal dynamics, specifically "trending analysis" to track how these fashion triplets evolve over time. Additionally, we aim to explore influence diffusion to understand how trends originate and spread across social networks.
Summary updated