Learning with Memory-based Virtual Classes for Deep Metric Learning

ByungSoo Ko, Geonmo Gu, Han-Gyu Kim

Abstract

Deep Metric Learning (DML) aims to learn an embedding space that generalizes to unseen classes. However, models often overfit to training categories, limiting their performance on novel data. We present MemVir (Memory-based Virtual classes), a novel training strategy that utilizes past embedding features and class weights as "virtual classes." By treating these historical representations as distractors and employing a curriculum learning schedule, MemVir forces the model to learn more robust and generalized features. MemVir is a training-only modification that adds no inference overhead and achieves state-of-the-art results on standard image retrieval benchmarks.

Introduction

The core objective of Deep Metric Learning (DML) is to map data into a feature space where semantic similarity corresponds to geometric distance. Unlike standard classification, DML's success is measured by its ability to generalize to previously unseen categories. A persistent challenge is the "strong focus" problem, where models over-optimize for the specific boundaries of training classes, leading to poor generalization.

While existing memory-based methods often use queues to increase the number of samples per class, they do not fundamentally change the label space. MemVir addresses this by introducing "virtual classes" derived from previous training iterations. These virtual classes act as semantic distractors, preventing the embedding space from collapsing too tightly around training samples and encouraging a more "stretched" and robust representation.

Methodology

MemVir introduces two primary innovations: memory-based virtual classes and a curriculum-based scheduling strategy.

Memory-based Virtual Classes

MemVir maintains two first-in-first-out (FIFO) queues: an embedding queue $\mathbb{E}$ and a weight queue $\mathbb{W}$. At each training step, the current mini-batch embeddings and class weights are stored. Instead of augmenting existing classes, MemVir treats data from past steps as entirely new "virtual classes."

The MemVir architecture utilizes memory queues for embeddings and weights to create virtual classes during training.

As shown in the figure above, the total number of classes effectively increases from $C$ to $(N+1)C$, where $N$ is the number of historical steps utilized. This expansion forces the model to distinguish between current "actual" classes and their slightly different historical "virtual" counterparts. The hyperparameter $M$ (the step gap between stored virtual classes) ensures that these virtual classes are sufficiently distinct from current representations to provide a meaningful challenge.

Curriculum Learning Schedule

Introducing a large number of virtual classes at the start of training can destabilize optimization. To mitigate this, MemVir employs a two-phase curriculum:

  1. Warm-up: The model is initially trained only on actual classes until a stable embedding structure begins to form.
  2. Step-pacing: Virtual classes are introduced incrementally from the memory queue, gradually increasing the task's difficulty.

This strategy ensures that the model first learns the basic class structures before being challenged by the more difficult task of distinguishing them from historical distractors.

Experimental Results

The authors evaluated MemVir on standard DML benchmarks, including CUB-200-2011, CARS196, and Stanford Online Products (SOP), using the Metric Learning Reality Check (MLRC) protocol.

Performance Gains

MemVir consistently improves performance across various loss functions. For instance, on the CUB-200 dataset, applying MemVir to a Proxy-NCA baseline increased Recall@1 from 64.3% to 68.3%. On CARS196, MemVir with ArcFace achieved a MAP@R of 29.63%, significantly outperforming the 27.22% baseline.

Performance comparison (Recall@1) over training steps, highlighting the impact of MemVir and its scheduling components.

The figure above demonstrates the effectiveness of the proposed scheduling. Turning on MemVir at the $U$-th step leads to a clear performance boost compared to the baseline. Furthermore, the ablation study shows that both the warm-up phase and step-pacing are critical for achieving optimal results.

Robustness and Gradient Health

Mathematical analysis reveals that MemVir helps maintain healthy gradient magnitudes. In standard softmax-based training, gradients often vanish as the model becomes confident, leading to stagnation in local minima. The introduction of virtual classes provides a persistent "push," helping the model escape local optima. Additionally, MemVir-trained models exhibit greater robustness to input deformations like Gaussian noise or cutout, indicating more generalized feature representations.

Conclusion

MemVir offers a simple yet powerful training-only strategy to improve generalization in Deep Metric Learning. By leveraging historical data as virtual classes and employing a curriculum learning schedule, it effectively combats overfitting to training categories. Its compatibility with existing loss functions and zero inference overhead make it a highly practical addition to the DML toolkit.

Summary updated

Abstract

The core of deep metric learning (DML) involves learning visual similarities in high-dimensional embedding space. One of the main challenges is to generalize from seen classes of training data to unseen classes of test data. Recent works have focused on exploiting past embeddings to increase the number of instances for the seen classes. Such methods achieve performance improvement via augmentation, while the strong focus on seen classes still remains. This can be undesirable for DML, where training and test data exhibit entirely different classes. In this work, we present a novel training strategy for DML called MemVir. Unlike previous works, MemVir memorizes both embedding features and class weights to utilize them as additional virtual classes. The exploitation of virtual classes not only utilizes augmented information for training but also alleviates a strong focus on seen classes for better generalization. Moreover, we embed the idea of curriculum learning by slowly adding virtual classes for a gradual increase in learning difficulty, which improves the learning stability as well as the final performance. MemVir can be easily applied to many existing loss functions without any modification. Extensive experimental results on famous benchmarks demonstrate the superiority of MemVir over state-of-the-art competitors. Code of MemVir is publicly available.
BibTeX
@inproceedings{ko2021learning,
  author = {ByungSoo Ko and Geonmo Gu and Han-Gyu Kim},
  title = {{Learning with Memory-based Virtual Classes for Deep Metric Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01158},
}
Figure