TL;DR
This paper reframes Fashion Knowledge Extraction (FKE) from social media posts as a multimodal captioning problem rather than a traditional classification or pipeline task. By converting structured fashion tuples into natural language descriptions, the proposed generative framework captures deep cross-modal dependencies and significantly outperforms existing methods, especially in complex, multi-person scenarios.
What Problem Does This Paper Address?
Every day, millions of fashion-related posts containing both images and text are uploaded to social media. Extracting structured "fashion knowledge"—such as the occasion, person attributes (age, gender), and specific clothing items—is incredibly valuable for personalized recommendation and image retrieval.
However, existing Fashion Knowledge Extraction (FKE) methods face major bottlenecks:
- Studio vs. Social Media: Most models are trained on clean, studio product images, failing to leverage the rich, informal contextual text found in social media posts.
- Rigid Classification: Traditional methods treat attribute extraction as a classification problem, requiring a fixed set of predefined categories. This fails to capture highly varied descriptions (e.g., "muslin white" vs. "chiffon beige").
- Error Propagation: Pipeline-based models first detect bounding boxes for people and then classify their clothes. If the initial person detection fails, the entire downstream extraction is ruined.
Furthermore, traditional models classify attributes independently, missing crucial context (e.g., knowing the occasion is a "wedding" provides a strong prior that a woman might be wearing a "white dress").
Methods
The authors introduce an end-to-end generative framework that treats FKE as an image-captioning task.
1. Tuples as Captions
Instead of predicting structured tuples directly, the authors design a deterministic Caption Construction strategy (Algorithm 1). This algorithm converts structured tuples like (wedding, female, youth, dress, chiffon beige) into a coherent natural language pseudo-caption: "The occasion is wedding. The first youth female wears a chiffon beige dress."
2. Model Architecture
The core model is built on VL-BART, a pre-trained multimodal encoder-decoder network.
- Input Encoding: The image is converted into Region of Interest (ROI) features using Faster R-CNN, while the post text is combined with a task-specific prefix. Both are processed by a bidirectional multimodal encoder.
- Autoregressive Decoding: The text decoder generates the pseudo-caption token-by-token. Because it is autoregressive, generating early tokens (like the occasion) naturally guides the generation of later tokens (the clothing items).
- Tuple Recovery: At inference time, the generated natural language caption is parsed back into structured fashion tuples using simple string-splitting rules.
3. Fashion-Specific Auxiliary Tasks
To warm up the pre-trained model before the main captioning task, the authors introduce three multi-task auxiliary objectives:
- Sentence Reconstruction (SRC): Masking 30% of the fashion caption and forcing the model to fill in the blanks based on visual context.
- Image-Text Matching (ITM): A binary generation task where the model predicts "true" or "false" based on whether a fashion caption matches the image.
- Visual Question Answering (VQA): Asking the model specific questions about the occasion, person attributes, or item appearances to anchor its fashion vocabulary.
Datasets
Because no public multimodal dataset for social media FKE existed, the authors collected and annotated a new benchmark:
- FKE Dataset: Consists of 9,272 Instagram posts across six occasions (school, graduation, sports, wedding, daily wear, vacation).
- Annotations: Professionally annotated by 10 fashion experts, yielding 32,439 fashion knowledge tuples (an average of 3.5 tuples per post and 2.7 tuples per person).
Key Insights and Ideas
- The Power of Generation over Discrimination: Generative models excel at handling "rare" or "unseen" attribute combinations. While a discriminative classifier struggles if it hasn't seen a specific label combination frequently, a generative decoder can flexibly compose descriptions out of its learned vocabulary.
- Exploiting Contextual Priors: By forcing the model to generate the occasion first, the autoregressive decoder establishes a semantic theme that improves the accuracy of subsequent fashion item predictions.
- Text Matters: Ablation studies showed that removing the user's post text caused a massive drop in performance, proving that social media text provides vital clues about context and intent that images alone cannot convey.
Key Results
The captioning approach achieved a new state-of-the-art on the FKE dataset, dramatically outperforming classification-based pipelines:
- Overall Performance: The proposed model achieved an Overall F1 score of 35.4, vastly outperforming the previous state-of-the-art pipeline model, HDF (21.6 F1), and the strong multimodal classification baseline, ViLBERT (23.5 F1).
- Rare and Unseen Data: On rare fashion tuples (appearing $<5$ times in training), the model achieved a 21.22% recall rate, compared to ViLBERT's 6.21% and HDF's 6.07%.
- Multi-Person Complexity: As the number of people and fashion items in an image increased, the performance gap between the author's model and pipeline models widened significantly, demonstrating its resilience against error propagation.
Limitations and Future Work
While highly effective, the paper notes a few areas for improvement:
- Format Dependency: If the decoder generates a caption that violates the expected template structure during inference, the parsing step fails, and the prediction is treated as null.
- Object Detector Reliance: The model still relies on an external, frozen object detector (Faster R-CNN) to extract image regions, which may limit its ability to catch highly fine-grained or unusual fashion accessories.
Future work could explore fully end-to-end vision-language models that bypass regional object detectors entirely.
Summary updated