"Does it come in black?" CLIP-like models are zero-shot recommenders

Patrick John Chia, Jacopo Tagliabue, Federico Bianchi, Ciro Greco, Diogo GonçalvesarXiv 2022 · DOI · arXiv

Abstract

Traditional e-commerce recommendation systems excel at finding substitutes or complements but struggle with comparative discovery—the ability to navigate a product catalog along specific semantic dimensions (e.g., "shorter" or "darker"). This paper introduces GradREC, a zero-shot recommendation framework that leverages the latent space of FashionCLIP to enable natural language "gradients." By identifying semantic directions within a multi-modal embedding space, GradREC allows users to discover products through relative adjustments without requiring explicit attribute labels or behavioral training data.

Introduction

When a shopper views a white dress, a standard recommender might suggest other white dresses or matching accessories. However, if that shopper wants the same style but in a darker shade or with a higher heel, traditional systems often fail. These systems are typically "static," relying on fixed metadata or historical clickstream data. To support comparative discovery, developers usually have to build expensive, supervised models for every possible attribute—a process that is brittle and difficult to scale across the vast, ever-changing landscape of fashion.

GradREC addresses this by treating recommendation as a traversal through a learned multi-modal latent space. By utilizing the geometric properties of models like CLIP, the system enables complex product discovery using simple natural language prompts.

The Problem of Comparative Discovery

Most modern e-commerce systems are built on similarity or co-occurrence. While effective for "more like this" scenarios, they lack a fundamental understanding of the relationships between attributes. A system might know that two items are "blue," but it rarely understands the gradient between "light blue" and "navy" unless specifically programmed with those labels. Building such systems requires massive amounts of human-annotated data, which is often unavailable for niche or emerging fashion trends.

Methodology: The GradREC Framework

GradREC builds upon FashionCLIP, a version of OpenAI’s CLIP architecture fine-tuned on over 800,000 fashion products from the Farfetch platform. The framework operates through two primary mechanisms:

1. The Traversal Vector ($v_c$)

To move along a semantic dimension (like "darker"), the model must first define that direction. GradREC uses zero-shot retrieval to identify a set of "neutral" items (e.g., "blue shirt") and "exemplar" items (e.g., "dark blue shirt"). It then employs a channel-wise Signal-to-Noise Ratio (SNR) method to isolate the specific dimensions in the 512-dimensional embedding space that encode the difference between these sets. This creates a vector representing the desired "gradient."

2. The Traversal Function ($\Phi$)

Once the direction is established, the model shifts the original product's embedding vector along that path. However, moving in a straight line through high-dimensional space can lead to "dead zones" where no real products exist. To prevent this, GradREC includes a regularization term based on the mean of the current point’s nearest neighbors. This ensures the "walk" stays within the manifold of realistic products, maintaining the stylistic integrity of the original item while modifying the target attribute.

Data and Implementation

The researchers utilized the FashionCLIP dataset, comprising over 800,000 products from the global luxury platform Farfetch. This dataset includes high-resolution images and rich natural language descriptions. By releasing the fine-tuned weights and code, the authors provide a foundation for the community to implement zero-shot discovery without the need for massive local compute resources or proprietary labeling.

Key Insights

  • Semantic Topology: Over-parameterized models like CLIP naturally organize concepts geometrically. The latent space inherently understands gradients (e.g., short to long) even without explicit instruction on those specific dimensions.
  • Zero-Shot Directionality: Vector arithmetic on natural language queries can induce directions that the model can follow, eliminating the need for per-attribute training.
  • Style Preservation: Unlike simple keyword filters, GradREC tends to preserve the "vibe" or aesthetic of the starting item. It changes the requested dimension (e.g., heel height) while attempting to keep the material, color, and brand aesthetic consistent.

Results and Evaluation

Qualitative analysis demonstrates that GradREC successfully navigates dimensions such as luminance, heel height, and garment length. For instance, starting with a light blue polo, the model can iteratively retrieve increasingly darker versions in a step-by-step fashion.

Quantitatively, GradREC outperforms standard visual similarity (k-nearest neighbors) in discovery tasks. In simulations for "blue shirts," GradREC produced distinct peaks in retrieval for light, medium, and dark items in the correct order. In contrast, standard kNN failed to capture this directionality, often returning items that were visually similar but lacked the requested semantic shift.

Limitations and Future Work

Despite its success, GradREC faces several challenges:

  • Retrieval Sensitivity: The accuracy of the traversal depends on the quality of the initial zero-shot retrieval used to define the direction. Noisy exemplar sets result in inaccurate gradients.
  • The "Formality" Trap: The model can be susceptible to dataset biases. When asked for a "more formal" version of a pink sandal, the model sometimes steered toward black leather dress shoes. This suggests the model's concept of "formal" is heavily tied to specific materials and colors present in the training data.
  • Monotonicity and Uncertainty: There is currently no mathematical guarantee that every step in a traversal is "more" of an attribute than the last. Furthermore, the model does not yet signal uncertainty when asked to apply a gradient that may not make sense for a specific item.

Conclusion

GradREC represents a significant step toward more intuitive, conversational e-commerce. By leveraging the latent geometry of multi-modal models, it turns recommendation from a static lookup into a dynamic exploration, allowing users to navigate product catalogs with the same nuance they use in natural language.

Summary updated

BibTeX
@misc{chia2022does,
  author = {Patrick John Chia and Jacopo Tagliabue and Federico Bianchi and Ciro Greco and Diogo Gonçalves},
  title = {{"Does it come in black?" CLIP-like models are zero-shot recommenders}},
  year = {2022},
  eprint = {2204.02473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.02473},
}
Figure