Deep Factorized Metric Learning

Chengkun Wang, Wenzhao Zheng, Junlong Li, Jie Zhou, Jiwen Lu

Abstract

Deep Metric Learning (DML) typically relies on a single backbone network to extract features, which often results in a "jack of all trades, master of none" model that learns generic features. We present Deep Factorized Metric Learning (DFML), a novel approach that factorizes the neural network backbone into specialized sub-blocks. By employing learnable routers within a Vision Transformer (ViT) architecture, DFML dynamically selects optimal processing paths for different image types. This method achieves state-of-the-art retrieval performance on standard benchmarks while simultaneously reducing computational costs by up to 24% FLOPs.

Introduction

Deep Metric Learning (DML) is fundamental to tasks like image retrieval and face recognition, where the goal is to map images into an embedding space where semantic similarity corresponds to Euclidean distance. Modern DML often uses ensemble methods to capture diverse features; however, these ensembles typically share a common backbone. This shared structure creates a bottleneck where the backbone must reconcile diverse training signals, leading to suboptimal, generic representations.

DFML addresses this by introducing structural diversity directly into the backbone. Instead of a monolithic network, the backbone is decomposed into multiple specialized sub-blocks, allowing the model to learn more nuanced and category-specific features without the overhead of traditional ensemble methods.

Methodology

Metric Factorization and Routing

DFML factorizes the core components of a Vision Transformer—the Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) layers—into $K$ smaller sub-blocks. For each layer, a lightweight, learnable router analyzes the input features and assigns the image to a specific sub-block.

The DFML architecture routes different image samples through specialized MHSA and MLP sub-blocks across T layers, creating unique processing paths.

This routing mechanism creates a vast space of $K^T$ potential paths through the network. Crucially, because each image only activates one sub-block per layer, the model maintains high efficiency despite its increased representational capacity.

Training Objectives

The model is trained using a dual-objective approach:

  1. Significance Loss ($J_s$): Routers are trained via a reconstruction objective to select the sub-block that best preserves the information of the original, unfactorized block.
  2. Discriminative Learning: The sub-blocks themselves are optimized using standard DML losses (e.g., ProxyAnchor or Triplet loss) to ensure the resulting embeddings are effective for similarity measurement.

Experimental Results

Retrieval Performance

DFML was evaluated on three major benchmarks: CUB-200-2011 (birds), Cars196 (cars), and Stanford Online Products (SOP). The model consistently outperformed existing state-of-the-art DML methods. Notably, on the Cars196 dataset, DFML achieved a 1.8% improvement in Recall@1 over baseline models.

Efficiency Gains

Beyond accuracy, DFML offers significant computational advantages. When applied to a ViT-Base architecture on ImageNet-1K, the factorized model achieved a +0.2% accuracy boost while requiring 24% fewer floating-point operations (FLOPs) compared to the standard architecture. This demonstrates that structural factorization can lead to both better performance and higher efficiency.

Qualitative Analysis of Specialization

A key insight of DFML is the natural specialization of sub-blocks. Qualitative analysis reveals that different sub-blocks focus on distinct semantic attributes. For instance, MHSA sub-blocks tend to specialize in capturing object pose and orientation, while MLP sub-blocks focus more on color and texture.

Qualitative results show that different sub-blocks specialize in distinct visual features, such as car orientation or color profiles.

Conclusion

Deep Factorized Metric Learning represents a shift from ensemble-based diversity to structural diversity within the backbone. By factorizing the network and utilizing intelligent routing, DFML breaks the "generic feature" bottleneck of traditional DML. The resulting models are not only more accurate but also more efficient, providing a scalable framework for high-performance metric learning across various domains.

Summary updated

Abstract

Learning a generalizable and comprehensive similarity metric to depict the semantic discrepancies between images is the foundation of many computer vision tasks. While existing methods approach this goal by learning an ensemble of embeddings with diverse objectives, the backbone network still receives a mix of all the training signals. Differently, we propose a deep factorized metric learning (DFML) method to factorize the training signal and employ different samples to train various components of the backbone network. We factorize the network to different sub-blocks and devise a learnable router to adaptively allocate the training samples to each sub-block with the objective to capture the most information. The metric model trained by DFML capture different characteristics with different sub-blocks and constitutes a generalizable metric when using all the sub-blocks. The proposed DFML achieves state-of-the-art performance on all three benchmarks for deep metric learning including CUB-200-20ll, Cars196, and Stanford Online Products. We also generalize DFML to the image classification task on ImageNet-1K and observe consistent improvement in accuracy/computation trade-off. Specifically, we improve the performance of ViT-B on ImageNet (+0.2% accuracy) with less computation load (-24% FLOPs). 1 1 Code is available at: https://github.com/wangck20/DFML.
BibTeX
@inproceedings{wang2023deep,
  author = {Chengkun Wang and Wenzhao Zheng and Junlong Li and Jie Zhou and Jiwen Lu},
  title = {{Deep Factorized Metric Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00741},
}

Loading related content...

Figure