Publisher Correction: Contrastive language and vision learning of general fashion concepts

Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Nunes Gonçalves, Ciro Greco, Jacopo TagliabueScientific 2023 · DOI · OpenAlex

Abstract

General-purpose vision-language models frequently struggle with the specialized vocabulary and fine-grained visual distinctions inherent to the fashion industry. This work presents FashionCLIP, a domain-specific adaptation of contrastive vision-language learning. By training on fashion-centric datasets, the model enables sophisticated multi-modal retrieval, zero-shot classification, and emergent object localization. This summary, based on a recent publisher correction, highlights the core architectural components and the model's ability to generalize to complex fashion concepts without task-specific fine-tuning.

Introduction

The fashion domain presents a unique challenge for computer vision: a "visual grammar" where subtle differences in texture, brand, and silhouette carry significant meaning. General models like OpenAI's CLIP, while powerful, often fail to distinguish between highly specific items such as a "Nike Tank Top" versus an "Adidas Dress." This research addresses this gap by developing FashionCLIP, a framework designed to understand general fashion concepts across text and image modalities.

Methodology

FashionCLIP utilizes a contrastive learning architecture consisting of dual encoders—one for text and one for images. These encoders map inputs into a shared embedding space where semantically similar concepts (e.g., the text "Yellow Sneakers" and an image of yellow sneakers) are positioned closely together. This allows for high-accuracy retrieval and classification without the need for massive, manually labeled datasets for every new trend.

Zero-Shot Capabilities

A primary strength of FashionCLIP is its zero-shot versatility. Instead of requiring a fixed set of labels during training, the model can classify images against arbitrary text labels provided at inference time. By calculating the similarity between an image embedding and various candidate text embeddings, the model selects the highest probability match.

Figure 2: Zero-shot classification workflow. The model compares an image embedding against multiple text labels (e.g., "Black Nike Dress") to identify the correct category via an argmax operation.

Emergent Localization

Beyond classification, the model demonstrates an "emergent" ability to localize objects. Despite not being explicitly trained on bounding box coordinates, the internal attention mechanisms of the contrastive model can be leveraged to generate precise localization maps. The model can produce accurate bounding boxes for items like straw hats and backpacks in complex scenes, purely as a byproduct of its multi-modal training.

Figure 7: Zero-shot localization in the wild. The model accurately identifies and bounds a "straw hat" in a complex scene from the Unsplash Lite dataset, a capability that emerges from multi-modal pre-training.

Key Results and Impact

Evaluations on datasets like ModaNet and Unsplash Lite demonstrate that domain specificity significantly improves performance in fashion-related tasks. FashionCLIP successfully bridges the gap between consumer language and visual representation, providing a robust foundation for next-generation fashion search engines and automated categorization tools. The model's ability to handle tasks it wasn't explicitly trained for—such as identifying a specific brand's apparel—highlights the potential of multi-modal pre-training to reduce reliance on manual labeling in specialized industries.

Summary updated

BibTeX
@inproceedings{chia2023publisher,
  author = {Patrick John Chia and Giuseppe Attanasio and Federico Bianchi and Silvia Terragni and Ana Rita Magalhães and Diogo Nunes Gonçalves and Ciro Greco and Jacopo Tagliabue},
  title = {{Publisher Correction: Contrastive language and vision learning of general fashion concepts}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-022-26364-y},
}

Loading related content...

Figure