A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation

Ling Xiao, Toshihiko YamasakiarXiv 2026 · arXiv

TL;DR

MCL-FIR introduces a multihead continual learning framework for fine-grained fashion image retrieval, allowing models to seamlessly learn new clothing attributes sequentially without forgetting old ones. By replacing costly triplet sampling with doublet-based contrastive learning and incorporating exponential moving average (EMA) distillation, it matches state-of-the-art static performance while slashing training costs by 70%.

What Problem Does This Paper Address?

Fine-grained Fashion Image Retrieval (FIR) is crucial for e-commerce, copyright protection, and recommendation systems, as it helps distinguish subtle visual differences (such as varying skirt lengths or specific collar designs). However, existing state-of-the-art models operate under a static learning assumption: whenever a new fashion attribute or trend emerges, the entire model must be retrained from scratch on all data combined.

This full retraining process is incredibly computationally expensive—some top-performing static methods require over 120 hours of training on a single high-end GPU for just one dataset. While large pre-trained vision-language models can perform zero-shot retrieval, their accuracy drops significantly without fine-tuning. Furthermore, parameter-efficient methods like prompt tuning fail to learn new attribute-specific visual features because their underlying image encoders remain frozen. This paper addresses the lack of a scalable, class-incremental learning (CIL) framework capable of efficiently adapting to evolving fashion attributes over time.

Fig. 1. The difference between MCL-FIR and SOTA static learning methods.

Methods

The authors propose MCL-FIR (Multihead Continual learning framework for fine-grained Fashion Image Retrieval), which introduces three main innovations:

  1. Lightweight Multi-head Attention Architecture: Instead of using a single-head model that suffers from feature interference when distributions shift, MCL-FIR shares a backbone image encoder (ResNet-50) but adds a lightweight, task-specific attention module ($\text{Attn}_i$) for each new attribute. This isolates attribute-specific knowledge and allows modular updates.
  2. Triplet-Free Doublet Learning via InfoNCE: Traditional fine-grained retrieval relies heavily on triplet loss, which requires complex, unstable, and computationally heavy anchor-positive-negative mining. The authors reformulate this into doublet sampling (only anchor and positive pairs) optimized via an InfoNCE contrastive loss, cutting data pipeline complexity and reducing training computation by a third.
  3. Exponential Moving Average (EMA) Distillation: To prevent the shared image encoder from forgetting previously learned features during incremental updates, the framework maintains an EMA "teacher" model. The teacher provides stable temporal supervision via a Mean Squared Error (MSE) distillation loss ($L_{kd}$), balancing plasticity (learning new attributes) and stability (retaining old ones).

Datasets

The framework was evaluated across four distinct datasets to simulate real-world sequential learning scenarios:

  • FashionAI: 180,335 images with 8 fine-grained attributes (e.g., skirt-length, collar-design).
  • DeepFashion: 289,222 images covering 6 attributes and 1,050 sub-classes.
  • DARN: A dataset with 9 attributes characterized by data sparsity due to incomplete image links.
  • Zappos50K: A footwear dataset used specifically to test the model's resilience against extreme semantic shifts (moving from apparel to shoes).

Key Insights and Ideas

  • Doublets over Triplets: Shifting from triplet ranking to instance-level contrastive learning on doublets provides a cleaner optimization landscape for incremental learning. This avoids the representation collapse often caused by bad triplet mining in dynamic data streams.
  • Zero Inference Routing Overhead: Because the query attribute explicitly dictates which lightweight attention head to activate at inference time, MCL-FIR completely bypasses the complex task-routing problems that typically plague classical class-incremental learning systems.
  • Cross-Domain Complementarity: Interestingly, learning fine-grained apparel attributes early on actually helps the model learn subsequent tasks better by building a richer, shared foundational representation in the image encoder.

Key Results

  • Massive Efficiency Gains: MCL-FIR achieves comparable retrieval accuracy to state-of-the-art static methods while consuming only ~30% of the total training time (e.g., 65.32 hours total across datasets compared to 469.86 hours for the static RPF model).
  • Outperforming CIL Baselines: It significantly outperforms standard incremental baselines like Experience Replay (ER) and standard multi-head triplet models. For example, on FashionAI, MCL-FIR achieved a mean Average Precision (mAP) of 64.41%, compared to just 22.08% for the ER baseline and 24.09% for the multi-head triplet baseline.
Fig. 4. We implemented two CIL baselines for comparison: an ER method and a multi-head method.
  • Resilience to Semantic Shift: When sequentially trained across apparel and then transitioned to shoes (Zappos50K), the model maintained high performance on previous tasks with virtually zero catastrophic forgetting.

Limitations and Future Work

While MCL-FIR offers a dramatic improvement in efficiency and scalability, the authors note that static training (retraining everything from scratch) can still achieve slightly higher peak accuracy in certain edge cases. Future work will focus on enriching visual representations using additional forms of supervision and exploring advanced continual learning strategies like dynamic model expansion or stricter parameter isolation.

Summary updated

Abstract

Most fine-grained fashion image retrieval (FIR) methods assume a static setting, requiring full retraining when new attributes appear, which is costly and impractical for dynamic scenarios. Although pretrained models support zero-shot inference, their accuracy drops without supervision, and no prior work explores class-incremental learning (CIL) for fine-grained FIR. We propose a multihead continual learning framework for fine-grained fashion image retrieval with contrastive learning and exponential moving average (EMA) distillation (MCL-FIR). MCL-FIR adopts a multi-head design to accommodate evolving classes across increments, reformulates triplet inputs into doublets with InfoNCE for simpler and more effective training, and employs EMA distillation for efficient knowledge transfer. Experiments across four datasets demonstrate that, beyond its scalability, MCL-FIR achieves a strong balance between efficiency and accuracy. It significantly outperforms CIL baselines under similar training cost, and compared with static methods, it delivers comparable performance while using only about 30% of the training cost. The source code is publicly available in https://github.com/Dr-LingXiao/MCL-FIR.
BibTeX
@misc{xiao2026multihead,
  author = {Ling Xiao and Toshihiko Yamasaki},
  title = {{A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation}},
  year = {2026},
  eprint = {2603.20648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.20648},
}
Figure