Multi-Proxy Learning from an Entropy Optimization Perspective

Yunlong Yu, Dingyi Zhang, Yingming Li, Zhongfei ZhangIJCAI 2022 · DOI

TL;DR

This paper introduces Multi-Proxy Learning (MPL), a novel approach for Deep Metric Learning that addresses the challenge of complex, non-uniform intra-class data distributions. MPL uses multiple proxies per class and optimizes an entropy-based objective to simultaneously encourage intra-class diversity and inter-class smoothness, leading to improved feature representations for downstream tasks.

What Problem Does This Paper Address?

Deep Metric Learning (DML) aims to learn an embedding space where similar samples are close and dissimilar ones are far apart. This is crucial for applications like face recognition and image retrieval. Traditional DML methods often struggle with "fine-grained" datasets where samples within the same class can exhibit significant visual variations (e.g., different poses, lighting, or sub-types). Existing proxy-based methods, which use a single proxy per class, can oversimplify these complex distributions, leading to poor local minima and limited generalization. This paper tackles this by proposing a method that can better capture and represent these intricate intra-class variations.

Methods

The core of MPL lies in its multi-proxy approach combined with an entropy optimization framework.

  • Multi-Proxy Learning: Instead of a single proxy per class, MPL learns a set of proxies for each class. This allows the model to represent different clusters or variations within a single class.
  • Multi-Proxy Cross-Entropy Loss: This loss function ensures that an instance is pulled towards its closest intra-class proxy while being pushed away from all other class proxies. This is a data-to-class approach that leverages the multi-proxy representation.
  • Entropy Optimization: The paper introduces two key regularization terms based on entropy:
  • Intra-class Diversity Regularization: This term minimizes the entropy of the conditional probability distribution of an instance with respect to its intra-class proxies. The goal is to encourage the proxies to be distinct and capture the diversity within a class, preventing "mode collapse" where proxies become too similar.
  • Inter-class Smoothness Regularization: This term maximizes the entropy of the class predictions for both instances and proxies. This encourages smoother probability distributions across classes, preventing overfitting and improving generalization, especially for visually similar classes.

The overall loss function combines the cross-entropy loss with these two entropy-based regularization terms, controlled by hyper-parameters $\alpha$ and $\beta$.

Datasets

The proposed MPL method was evaluated on three benchmark datasets:

  • CUB (Caltech-UCSD Birds-200-2011): A fine-grained dataset of bird species.
  • Cars196: A dataset of car classes.
  • Stanford Online Products (SOP): A dataset of product images.

The paper follows standard data pre-processing and augmentation techniques for these datasets.

Key Insights and Ideas

  • The "Proxy Collapse" Problem: The paper highlights that even with multiple proxies, simply assigning them can lead to them becoming too similar (mode collapse), failing to capture intra-class diversity. The intra-class diversity regularization directly addresses this.
  • Entropy as a Regularizer: The novel application of entropy to regularize both intra-class diversity and inter-class smoothness is a key contribution. It provides a principled way to manage the trade-off between discriminative power and generalization.
  • Bridging Proxy and Pair-Based Methods: The multi-proxy approach, by considering relationships between instances and multiple proxies, implicitly captures some of the rich pairwise information that traditional pair-based losses focus on, while retaining the efficiency of proxy-based methods.
  • Fine-Grained Data Demands Nuance: The success on CUB and Cars196, and the relative struggle on SOP (due to fewer samples per class), underscores that the effectiveness of multi-proxy learning is tied to the complexity and distribution of intra-class data.

Key Results

  • MPL achieves state-of-the-art or competitive performance on fine-grained image retrieval and clustering tasks on CUB and Cars196 datasets.
  • For example, using a ResNet50 backbone, MPL achieved 70.4% R@1 on CUB and 88.1% R@1 on Cars196 for image retrieval, outperforming previous methods.
  • The ablation studies show that increasing the number of proxies per class generally improves performance, up to a point (around 5 proxies for CUB/Cars196).
  • Both regularization terms are crucial; combining them yields the best results, demonstrating their complementary effects.
  • Larger input image sizes and higher embedding dimensionality generally lead to better performance.

Limitations and Future Work

  • SOP Dataset Performance: MPL did not achieve the best performance on the SOP dataset. The authors attribute this to the very small number of samples per class (averaging 5), which makes it difficult to establish a meaningful multi-center structure. This suggests that MPL is more effective when there's sufficient data to support multiple intra-class clusters.
  • Hyper-parameter Sensitivity: While the retrieval task was robust to hyper-parameters $\alpha$ and $\beta$, the clustering task showed some sensitivity. The paper suggests that encouraging intra-class proxies to be distributed (increasing $\beta$) generally benefits performance.
  • Future Directions: The paper implicitly suggests exploring how to adapt MPL to datasets with even fewer samples per class or investigating more sophisticated ways to determine the optimal number of proxies per class. Further research could also explore different entropy-based objectives or their combination with other DML techniques.

Summary updated

BibTeX
@inproceedings{yu2022multi,
  author = {Yunlong Yu and Dingyi Zhang and Yingming Li and Zhongfei Zhang},
  title = {{Multi-Proxy Learning from an Entropy Optimization Perspective}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/222},
}
Figure