Pre-Training Street-to-Shop Retrievers on Synthetic Street Photos of the Catalog

When a shopper photographs a dress on the street and asks a store to find it, the system has to match a blurry, cluttered, oddly lit snapshot to one clean studio image among hundreds of thousands. Current systems learn this match almost entirely from hand-labeled street-to-shop pairs, starting from backbones pre-trained on generic photos (ImageNet). Text retrieval faced a similar problem and found a better route. Generic pre-training does not teach a model the specific relationship between a short, messy query and the document it should find. Pre-training on large volumes of synthetic query–document pairs shaped like the real task does teach it. We propose to carry that lesson into visual retrieval. Every retailer already owns its catalog, and catalog images are easy to cut out from their white backgrounds. We use them to manufacture millions of fake "street photos": garments composited onto clutter, warped, partially hidden, blurred, shrunk, and relit. We then pre-train a retriever to find the original catalog image from each fake. No human labels are needed before the final fine-tuning step.

The second part of the idea follows from how retrieval is deployed. Catalog images are encoded once, offline, so the encoder that embeds them can be large. Shopper queries are encoded live, so the encoder that embeds them should be small. We therefore pre-train a compact query encoder to land directly in the embedding space of a larger catalog encoder. We test whether matched pre-training lets this small, specialized query encoder match or beat a full-size symmetric model, the visual counterpart of what has been reported for compact text encoders in e-commerce search.

Core Claim

A pre-training task that reproduces the structure of consumer-to-shop retrieval closes more of the street-to-shop gap than either generic self-supervised augmentation or additional labeled pairs, at equal compute and label budgets. The task pairs a synthetic, degraded view of a catalog item with its clean catalog image, retrieved against the full catalog with hard negatives mined from it. We further claim that this pre-training is what allows an asymmetric design, with a small query encoder mapping into a frozen large catalog space, to work.

The claim stands or falls on one contrast. If ordinary random crops and color jitter on the same catalog images do as well as our consumer-matched corruptions and retrieval objective, the idea reduces to "self-supervised learning on in-domain images." We would report that result rather than argue around it.

Where This Sits

The method comes from text retrieval. Domain-matched Pre-training Tasks for Dense Retrieval pre-trains two-tower retrievers on 65 million synthetic questions and adds iterative hard-negative mining, which uses the model's own false positives as negatives. It reports 84.0% top-20 accuracy on Natural Questions. It also found that RoBERTa-large beat BERT-base only after domain-matched pre-training. Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search builds fake queries by sampling random substrings of product titles and pre-aligns query and title towers. A 4-layer model trained this way beats a 12-layer general BERT on long-tail queries. Both papers are text-only. We reuse their recipe of a synthetic query generator, a retrieval objective, and iterative hard negatives. Image degradation replaces substring sampling, and the asymmetric-capacity question is posed for vision.

The motivation and the constraints come from visual retrieval. Where to Buy It shows that learned similarity beats whole-image search on Exact Street2Shop (37.1% vs 22.2% top-20 on dresses). It also shows a collapse on small items (9.6% on footwear) and finds that humans pick the exact match among 10 candidates only 54% of the time. Deep Metric Learning for Cross-Domain Fashion Instance Retrieval gives the strongest metric-learning numbers on DeepFashion2 and reports two findings that constrain our design. Consumer-to-shop retrieval beats the reverse direction by 2–4%. Graph diffusion re-ranking hurts when there are many identities with few images each, so we do not rely on post-hoc re-ranking. DeepFashion2 shows that Match R-CNN retrieval falls from 0.617 to 0.564 top-20 when ground-truth boxes are replaced by detected ones, and that detection collapses under heavy occlusion. Video2Shop argues for separate branches per domain. Closing the Gap Between Query and Database through Query Feature Transformation maps query features into gallery space with a lightweight transform on top of a fixed backbone. That transform is the simplest version of our asymmetric idea, and we treat it as a baseline, not a precedent to be absorbed.

The closest connecting work, Large Language Models as Visual Cross-Domain Learners (VLLaVO), bridges visual style gaps by converting images to tags and captions and LoRA-tuning a 7B language model. It reaches 98.5% on PACS and 91.6% on OfficeHome adaptation. It differs from our proposal on every axis that matters here. It does category classification rather than exact-instance retrieval. It runs a multi-model generation pipeline that cannot index a 400k-image catalog. It adapts a general checkpoint rather than pre-training on task-shaped data. Its 30.1% gain over a zero-shot language model does support our premise that generic capacity alone does not bridge domain shift.

Research Program

We build a corruption pipeline that turns a catalog image $x$ into a synthetic consumer view $T(x)$. The garment is segmented, a trivial step on white-background studio shots. It is then composited onto backgrounds drawn from scene images and given non-rigid warps that mimic draping and folding. Further steps add occluders over a random fraction of the item, downscale it to a few percent of the frame (aimed at the footwear failure), apply motion blur and noise, and shift illumination and white balance. Where a listing has several images, such as on-model and flat shots, we also use sibling images as natural query views. These free pairs are closest in spirit to the flat-versus-fitted gap.

A query encoder $f$ embeds the corrupted view and a catalog encoder $g$ embeds clean images. Both are trained with a contrastive retrieval loss over the catalog:

$$\mathcal{L} = -\log \frac{\exp(f(T(x))^\top g(x)/\tau)}{\exp(f(T(x))^\top g(x)/\tau) + \sum_{x^- \in \mathcal{N}(x)} \exp(f(T(x))^\top g(x^-)/\tau)}$$

Here $\tau$ is a temperature and $\mathcal{N}(x)$ is a set of negatives. The negative set holds other items in the batch plus same-category near neighbors mined from the catalog. Following the DPR-PAQ schedule, a second round adds the model's own top-ranked false positives. Catalogs drawn from 25 stores (Exact Street2Shop) contain the same product under different listings. We therefore remove near-duplicates by image hashing and embedding distance before mining, so that true matches are not punished as negatives.

We study two architectures. The symmetric version uses ResNet50 for both towers, matching the backbone of the strongest DeepFashion2 metric-learning results. The asymmetric version keeps $g$ as that ResNet50 and replaces $f$ with a compact network of roughly a quarter of its inference cost. After pre-training, all models are fine-tuned on DeepFashion2 consumer–commercial pairs with the same N-pair loss used by the metric-learning baseline. This isolates the effect of pre-training from the effect of the fine-tuning loss.

We pre-train in two regimes. The inductive regime uses only DeepFashion2 training-split commercial images. The catalog-transductive regime also includes the unlabeled Exact Street2Shop shop photos. The second regime is legitimate for deployment, because a retailer always holds its own catalog, but we report it separately so that gains from seeing gallery images are visible.

Baselines

The naive reference point is an ImageNet-pretrained ResNet50 used with no cross-domain training, plain cosine search on cropped images. It shows how much of the gap any training closes. The strongest reported competitor on DeepFashion2 is the metric-learning study Deep Metric Learning for Cross-Domain Fashion Instance Retrieval. Its single N-pair ResNet50 model reaches Recall@20 = 0.648 on consumer-to-shop with ground-truth crops. Ensembling two N-pair models reaches 0.676. We re-run their single-model recipe ourselves on the same split (about 192k training and 32k validation images), with the same 128-dimensional embedding and the same fine-tuning epochs and tuning budget we give our own models. Beating it means a single pre-trained model exceeding our reproduction of the single N-pair model by more than the seed-to-seed spread. Beating the 0.676 ensemble with one model would be the stronger claim.

For end-to-end retrieval with detection we compare against Match R-CNN from DeepFashion2, reported at 0.617 top-20 with ground-truth boxes and 0.564 with detected boxes. We plug our encoders behind the same detector, so the comparison measures only the embedding.

On Exact Street2Shop the reference numbers are the per-category top-20 accuracies from Where to Buy It: 37.1% on dresses, 54.6% on skirts, and 9.6% on footwear. These come from AlexNet features and are a floor, not the main competitor. For this reason we also re-run the N-pair ResNet50 baseline there.

For the asymmetric question, the direct competitor is the query feature transformation of Closing the Gap Between Query and Database. In our version, a learned mapping on top of a frozen ResNet50 sends query features into the frozen catalog space and is trained on the same labeled pairs. We compare it with the compact query encoder at matched query-side compute.

Two ablations isolate the key ingredients. The first keeps the same images, compute, and contrastive objective but swaps our consumer-matched corruptions for generic self-supervised augmentations (random crops, color jitter, blur). The second keeps the corruptions but replaces catalog-mined hard negatives with in-batch negatives only.

No reported result was found on Lacuna for a CLIP-style vision-language backbone on DeepFashion2 or Exact Street2Shop. We will run one, fine-tuned with the same N-pair recipe, as an unreported but important reference.

Why not just…?

Why not just use generic self-supervised augmentation on the catalog? This is the real threat. Random crops and color jitter already produce "degraded views," and in-domain images alone might explain any gain. Nothing in the file rules this out. The matched-augmentation ablation settles it at equal compute. Our claim requires the consumer-matched pipeline to win overall. It also requires a per-condition pattern: wins should concentrate on DeepFashion2's heavy-occlusion and small-scale subsets and on Street2Shop footwear, where the specific corruptions target known failures. If the generic version ties, we report that domain-specific images, not domain-shaped tasks, carry the effect.

Why not just fine-tune on more real labeled pairs? DeepFashion2 already offers 873k consumer–commercial pairs, and a reviewer may expect labels to dominate. The text-retrieval evidence suggests otherwise. DPR-PAQ gained 3.2 top-20 points on Natural Questions over non-pretrained baselines even with full supervised fine-tuning. We test this directly with a label-efficiency curve, fine-tuning on 1%, 10%, and 100% of the training pairs with and without pre-training. If the gap vanishes at 100%, the method is a label-saving device rather than a ceiling-raiser, which still matters for new catalogs that have no labeled pairs.

Why not just bridge the gap through text, as VLLaVO does? Large Language Models as Visual Cross-Domain Learners works for categories, but exact-instance retrieval depends on details that captions drop: a particular print, button spacing, a hem shape. Its cascade of CLIP, GPT-3, BLIP, and a 7B model also cannot be run per query against hundreds of thousands of gallery items. We include a cheap probe, caption-to-caption retrieval with an off-the-shelf captioner, on a held-out subset to measure how much identity survives the text bottleneck.

Why not just learn a linear query-to-gallery transform on a frozen backbone? It is cheap and proven for the flat-versus-fitted gap. A linear map, however, can only move query features after the backbone has already mixed clutter and lighting into them. The matched-compute comparison against the compact pre-trained query encoder tells us whether the shift must be absorbed inside the network or can be corrected afterward.

Proposed Experiments

The primary test is consumer-to-shop Recall@20 on DeepFashion2 with ground-truth crops, comparing four conditions. These are pre-trained versus not pre-trained, consumer-matched versus generic augmentation, and symmetric ResNet50 versus compact query encoder. Each condition is run over three seeds with identical fine-tuning. We also report shop-to-consumer Recall@20. Pre-training only the consumer side of the relationship could widen the 2–4% directional gap rather than narrow it, and we want to know which happens.

The end-to-end test replaces ground-truth boxes with Match R-CNN detections and asks whether the drop from crops to detections (0.617 to 0.564 for Match R-CNN) shrinks. Our corruptions include loose, off-center crops, so we expect detector imprecision to cost less.

The transfer test pre-trains inductively, fine-tunes on DeepFashion2 only, and evaluates zero-shot on Exact Street2Shop per category. We then repeat with catalog-transductive pre-training to measure what access to the target catalog adds.

The capacity test asks whether the compact query encoder, after matched pre-training, reaches the symmetric ResNet50's Recall@20. We pair this with measured query latency. It also checks the DPR-PAQ observation in reverse: matched pre-training should help the small model more than the large one.

The proposal is confirmed if matched pre-training beats both the reproduced single N-pair model and the generic-augmentation ablation beyond seed variance, with gains concentrated where the corruptions predict. It is refuted if generic augmentation ties it, or if all advantages disappear at full label budget and full model size.