Multimodal Fashion Knowledge Extraction as Captioning

Yifei Yuan, Wenxuan Zhang, Yang Deng, Wai Lam

Introduction

Extracting structured fashion knowledge from social media is a significantly more complex task than analyzing professional studio photography. Social media posts are inherently multimodal, combining noisy images with informal text, and often feature multiple people in diverse, unconstrained settings. Traditional Fashion Knowledge Extraction (FKE)—the task of identifying attributes like "youth female" wearing a "white dress" at a "wedding"—has historically relied on rigid pipelines. These pipelines typically detect person or clothing bounding boxes first and then classify those regions into pre-defined categories.

The FKE process maps multimodal social media posts to structured fashion knowledge tuples.

However, these traditional methods suffer from two critical flaws: error propagation and rigidity. If an initial person-detector fails to identify a box, the entire extraction for that individual is lost. Furthermore, classification models struggle with the "long tail" of fashion—unique textures, niche styles, or specific color combinations that were not well-represented in a fixed training set.

Reimagining Extraction as Translation

To address these limitations, researchers have reimagined FKE as a multimodal captioning problem. Instead of treating extraction as a series of classification labels, the task is cast as a sequence-to-sequence generation problem. Structured data is converted into natural language "pseudo-captions" that describe the scene. For example, a tuple like (wedding, female, youth, dress, white) is transformed into the sentence: "The occasion is wedding. The first youth female wears a white dress."

This approach allows the model to leverage the semantic dependencies between attributes. By using language, the model can learn that the word "wedding" often correlates with "white dress," providing a more holistic understanding of the scene than independent classifiers could achieve.

Methodology: VL-BART and Auxiliary Tasks

The framework utilizes VL-BART, a multimodal transformer architecture designed to encode both image regions (extracted via Faster R-CNN) and the accompanying post text. To enhance the model's performance, the authors introduced three auxiliary training objectives to "warm up" the transformer before the main extraction task:

  1. Sentence Reconstruction (SRC): Predicting masked tokens within fashion descriptions.
  2. Image-Text Matching (ITM): Determining if a generated fashion caption accurately matches the visual content of the image.
  3. Visual Question Answering (VQA): Answering specific queries about the scene, such as "What is the first person wearing?"
The model architecture uses a multimodal encoder-decoder with auxiliary tasks to refine fashion knowledge recovery.

Once the model generates a descriptive caption, a simple rule-based parser converts the natural language back into structured tuples for downstream use in recommendation engines or trend analysis.

Experimental Results and Insights

To evaluate the framework, the authors introduced the Instagram Fashion Dataset, comprising 9,272 posts across six occasions (School, Graduation, Sports, Wedding, Daily, and Vacation), annotated with over 32,000 fashion knowledge tuples.

The generative model achieved a new state-of-the-art on the FKE task, reaching an F1 score of 35.4. This significantly outperformed the previous leading pipeline-based model, HDF (21.6), and other multimodal baselines like ViLBERT (23.5).

Crucially, the model demonstrated superior resilience in handling "unseen" or rare fashion items. In categories never encountered during training, the model achieved a 6.97% recall, nearly four times higher than the HDF baseline (1.79%). This suggests that by treating fashion as language, the model can compose descriptions of novel items based on its broader understanding of individual words and textures.

Conclusion

By framing fashion knowledge extraction as a multimodal captioning task, this research moves away from fragile, box-dependent pipelines toward a more flexible, language-centric approach. While the current method relies on specific sentence templates and carries higher computational costs than simple classifiers, it represents a significant step forward in capturing the nuanced interplay between text and imagery in social media fashion.

Summary updated

Abstract

Social media plays a significant role in boosting the fashion industry, where a massive amount of fashion-related posts are generated every day. In order to obtain the rich fashion information from the posts, we study the task of social media fashion knowledge extraction. Fashion knowledge, which typically consists of the occasion, person attributes, and fashion item information, can be effectively represented as a set of tuples. Most previous studies on fashion knowledge extraction are based on the fashion product images without considering the rich text information in social media posts. Existing work on fashion knowledge extraction in social media is classification-based and requires to manually determine a set of fashion knowledge categories in advance. In our work, we propose to cast the task as a captioning problem to capture the interplay of the multimodal post information. Specifically, we transform the fashion knowledge tuples into a natural language caption with a sentence transformation method. Our framework then aims to generate the sentence-based fashion knowledge directly from the social media post. Inspired by the big success of pre-trained models, we build our model based on a multimodal pre-trained generative model and design several auxiliary tasks for enhancing the knowledge extraction. Since there is no existing dataset which can be directly borrowed to our task, we introduce a dataset consisting of social media posts with manual fashion knowledge annotation. Extensive experiments are conducted to demonstrate the effectiveness of our model.
BibTeX
@inproceedings{yuan2023multimodal,
  author = {Yifei Yuan and Wenxuan Zhang and Yang Deng and Wai Lam},
  title = {{Multimodal Fashion Knowledge Extraction as Captioning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3624918.3625315},
}
Figure