Abstract
Deep Metric Learning (DML) aims to learn an embedding space that generalizes to unseen classes. However, models often overfit to training categories, limiting their performance on novel data. We present MemVir (Memory-based Virtual classes), a novel training strategy that utilizes past embedding features and class weights as "virtual classes." By treating these historical representations as distractors and employing a curriculum learning schedule, MemVir forces the model to learn more robust and generalized features. MemVir is a training-only modification that adds no inference overhead and achieves state-of-the-art results on standard image retrieval benchmarks.
Introduction
The core objective of Deep Metric Learning (DML) is to map data into a feature space where semantic similarity corresponds to geometric distance. Unlike standard classification, DML's success is measured by its ability to generalize to previously unseen categories. A persistent challenge is the "strong focus" problem, where models over-optimize for the specific boundaries of training classes, leading to poor generalization.
While existing memory-based methods often use queues to increase the number of samples per class, they do not fundamentally change the label space. MemVir addresses this by introducing "virtual classes" derived from previous training iterations. These virtual classes act as semantic distractors, preventing the embedding space from collapsing too tightly around training samples and encouraging a more "stretched" and robust representation.
Methodology
MemVir introduces two primary innovations: memory-based virtual classes and a curriculum-based scheduling strategy.
Memory-based Virtual Classes
MemVir maintains two first-in-first-out (FIFO) queues: an embedding queue $\mathbb{E}$ and a weight queue $\mathbb{W}$. At each training step, the current mini-batch embeddings and class weights are stored. Instead of augmenting existing classes, MemVir treats data from past steps as entirely new "virtual classes."
As shown in the figure above, the total number of classes effectively increases from $C$ to $(N+1)C$, where $N$ is the number of historical steps utilized. This expansion forces the model to distinguish between current "actual" classes and their slightly different historical "virtual" counterparts. The hyperparameter $M$ (the step gap between stored virtual classes) ensures that these virtual classes are sufficiently distinct from current representations to provide a meaningful challenge.
Curriculum Learning Schedule
Introducing a large number of virtual classes at the start of training can destabilize optimization. To mitigate this, MemVir employs a two-phase curriculum:
- Warm-up: The model is initially trained only on actual classes until a stable embedding structure begins to form.
- Step-pacing: Virtual classes are introduced incrementally from the memory queue, gradually increasing the task's difficulty.
This strategy ensures that the model first learns the basic class structures before being challenged by the more difficult task of distinguishing them from historical distractors.
Experimental Results
The authors evaluated MemVir on standard DML benchmarks, including CUB-200-2011, CARS196, and Stanford Online Products (SOP), using the Metric Learning Reality Check (MLRC) protocol.
Performance Gains
MemVir consistently improves performance across various loss functions. For instance, on the CUB-200 dataset, applying MemVir to a Proxy-NCA baseline increased Recall@1 from 64.3% to 68.3%. On CARS196, MemVir with ArcFace achieved a MAP@R of 29.63%, significantly outperforming the 27.22% baseline.
The figure above demonstrates the effectiveness of the proposed scheduling. Turning on MemVir at the $U$-th step leads to a clear performance boost compared to the baseline. Furthermore, the ablation study shows that both the warm-up phase and step-pacing are critical for achieving optimal results.
Robustness and Gradient Health
Mathematical analysis reveals that MemVir helps maintain healthy gradient magnitudes. In standard softmax-based training, gradients often vanish as the model becomes confident, leading to stagnation in local minima. The introduction of virtual classes provides a persistent "push," helping the model escape local optima. Additionally, MemVir-trained models exhibit greater robustness to input deformations like Gaussian noise or cutout, indicating more generalized feature representations.
Conclusion
MemVir offers a simple yet powerful training-only strategy to improve generalization in Deep Metric Learning. By leveraging historical data as virtual classes and employing a curriculum learning schedule, it effectively combats overfitting to training categories. Its compatibility with existing loss functions and zero inference overhead make it a highly practical addition to the DML toolkit.
Summary updated