Variational Continual Proxy-Anchor for Deep Metric Learning

Minyoung Kim, Ricardo Guerrero, Hai Xuan Pham, Vladimir PavlovicAISTATS 2022 · OpenReview

Abstract

The recent proxy-anchor method achieved outstanding performance in deep metric learning, which can be acknowledged to its data efficient loss based on hard example mining, as well as far lower sampling complexity than pair-based approaches. In this paper we extend the proxy-anchor method by posing it within the continual learning framework, motivated from its batch-expected loss form (instead of instance-expected, typical in deep learning), which can potentially incur the catastrophic forgetting of historic batches. By regarding each batch as a task in continual learning, we adopt the Bayesian variational continual learning approach to derive a novel loss function. Interestingly the resulting loss has two key modifications to the original proxy-anchor loss: i) we inject noise to the proxies when optimizing the proxy-anchor loss, and ii) we encourage momentum update to avoid abrupt model changes. As a result, the learned model achieves higher test accuracy than proxy-anchor due to the robustness to noise in data (through model perturbation during training), and the reduced batch forgetting effect. We demonstrate the improved results on several benchmark datasets.

Introduction

Deep Metric Learning (DML) is the process of training neural networks to map images into an embedding space where similar items are clustered together and dissimilar items are separated. The current state-of-the-art method, Proxy-Anchor (PA), utilizes "proxies"—representative vectors for each class—to make training more efficient than traditional pair-based methods.

However, PA has a significant hidden weakness: batch forgetting. Because the loss function is calculated based on the specific samples within a minibatch, the model can over-optimize for the current batch, losing the "big picture" learned from previous ones. This "proxy drift" is particularly problematic when training with small batches or limited data, as the proxies fail to maintain a stable representation of the global class distribution.

Methodology: Variational Continual Learning

To address this, researchers developed Variational Continual Proxy-Anchor (VCPA). This approach reframes DML by treating every training batch as a "task" in a continual learning framework. By applying Bayesian Variational Continual Learning, the authors introduced two major innovations:

  1. Stochastic Proxies: Instead of fixed points, proxies are treated as Gaussian probability distributions. During training, the model samples a "noisy" version of the proxy. This stochasticity acts as a powerful regularizer, forcing the network to be robust to small perturbations and improving generalization.
  2. Momentum via KL Divergence: To prevent forgetting, a Kullback–Leibler (KL) divergence term is added to the loss function. This term penalizes the model if the new proxy distributions drift too far from those learned in previous batches, effectively acting as a "memory" of historical data.
Comparison of Proxy-Anchor (PA) and Variational Continual Proxy-Anchor (VCPA) showing how VCPA prevents misclassification caused by proxy drift.

As shown in the figure above, standard PA (e) can lead to misclassification of unseen data because the proxies ($p^{new}$) drift significantly based on the local batch. VCPA (f) maintains more stable, representative proxies that better align with the true class boundaries.

Key Results

VCPA was evaluated on four standard DML benchmarks: CUB-200-2011, Cars-196, Stanford Online Products (SOP), and In-shop Clothes Retrieval.

  • Superior Accuracy: VCPA consistently outperformed the original PA and other top-tier methods. For instance, on the CUB dataset, it improved Recall@1 from 68.4% to 69.9%.
  • Robustness to Small Data: When training on only 10% of the CUB dataset, VCPA achieved a Recall@1 of 58.6%, significantly higher than the standard PA's 56.3%.
  • Stability: The model showed significantly less performance degradation as batch sizes were reduced, proving the effectiveness of the momentum-based KL-regularization.
  • Uncertainty Alignment: Qualitative t-SNE visualizations revealed that the learned variance of the proxy distributions correctly captured class diversity—visually varied classes resulted in "wider" proxy distributions.

Conclusion and Limitations

By treating minibatches as tasks in a continual learning setup, VCPA successfully mitigates the batch forgetting effect inherent in standard Proxy-Anchor methods. While the approach is roughly 38% slower per batch (528ms vs 383ms) due to the second-order Newton updates required for the proxies, the gains in robustness and accuracy make it a superior choice for high-performance metric learning. Future work may explore fully Bayesian treatments of the entire network to further enhance performance, albeit at a higher computational cost.

Summary updated

Abstract

The recent proxy-anchor method achieved outstanding performance in deep metric learning, which can be acknowledged to its data efficient loss based on hard example mining, as well as far lower sampling complexity than pair-based approaches. In this paper we extend the proxy-anchor method by posing it within the continual learning framework, motivated from its batch-expected loss form (instead of instance-expected, typical in deep learning), which can potentially incur the catastrophic forgetting of historic batches. By regarding each batch as a task in continual learning, we adopt the Bayesian variational continual learning approach to derive a novel loss function. Interestingly the resulting loss has two key modifications to the original proxy-anchor loss: i) we inject noise to the proxies when optimizing the proxy-anchor loss, and ii) we encourage momentum update to avoid abrupt model changes. As a result, the learned model achieves higher test accuracy than proxy-anchor due to the robustness to noise in data (through model perturbation during training), and the reduced batch forgetting effect. We demonstrate the improved results on several benchmark datasets.
BibTeX
@inproceedings{kim2022variational,
  author = {Minyoung Kim and Ricardo Guerrero and Hai Xuan Pham and Vladimir Pavlovic},
  title = {{Variational Continual Proxy-Anchor for Deep Metric Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=OiTNpZcu8qQ},
}
Figure