Abstract
Deep Metric Learning (DML) typically relies on a single backbone network to extract features, which often results in a "jack of all trades, master of none" model that learns generic features. We present Deep Factorized Metric Learning (DFML), a novel approach that factorizes the neural network backbone into specialized sub-blocks. By employing learnable routers within a Vision Transformer (ViT) architecture, DFML dynamically selects optimal processing paths for different image types. This method achieves state-of-the-art retrieval performance on standard benchmarks while simultaneously reducing computational costs by up to 24% FLOPs.
Introduction
Deep Metric Learning (DML) is fundamental to tasks like image retrieval and face recognition, where the goal is to map images into an embedding space where semantic similarity corresponds to Euclidean distance. Modern DML often uses ensemble methods to capture diverse features; however, these ensembles typically share a common backbone. This shared structure creates a bottleneck where the backbone must reconcile diverse training signals, leading to suboptimal, generic representations.
DFML addresses this by introducing structural diversity directly into the backbone. Instead of a monolithic network, the backbone is decomposed into multiple specialized sub-blocks, allowing the model to learn more nuanced and category-specific features without the overhead of traditional ensemble methods.
Methodology
Metric Factorization and Routing
DFML factorizes the core components of a Vision Transformer—the Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) layers—into $K$ smaller sub-blocks. For each layer, a lightweight, learnable router analyzes the input features and assigns the image to a specific sub-block.
This routing mechanism creates a vast space of $K^T$ potential paths through the network. Crucially, because each image only activates one sub-block per layer, the model maintains high efficiency despite its increased representational capacity.
Training Objectives
The model is trained using a dual-objective approach:
- Significance Loss ($J_s$): Routers are trained via a reconstruction objective to select the sub-block that best preserves the information of the original, unfactorized block.
- Discriminative Learning: The sub-blocks themselves are optimized using standard DML losses (e.g., ProxyAnchor or Triplet loss) to ensure the resulting embeddings are effective for similarity measurement.
Experimental Results
Retrieval Performance
DFML was evaluated on three major benchmarks: CUB-200-2011 (birds), Cars196 (cars), and Stanford Online Products (SOP). The model consistently outperformed existing state-of-the-art DML methods. Notably, on the Cars196 dataset, DFML achieved a 1.8% improvement in Recall@1 over baseline models.
Efficiency Gains
Beyond accuracy, DFML offers significant computational advantages. When applied to a ViT-Base architecture on ImageNet-1K, the factorized model achieved a +0.2% accuracy boost while requiring 24% fewer floating-point operations (FLOPs) compared to the standard architecture. This demonstrates that structural factorization can lead to both better performance and higher efficiency.
Qualitative Analysis of Specialization
A key insight of DFML is the natural specialization of sub-blocks. Qualitative analysis reveals that different sub-blocks focus on distinct semantic attributes. For instance, MHSA sub-blocks tend to specialize in capturing object pose and orientation, while MLP sub-blocks focus more on color and texture.
Conclusion
Deep Factorized Metric Learning represents a shift from ensemble-based diversity to structural diversity within the backbone. By factorizing the network and utilizing intelligent routing, DFML breaks the "generic feature" bottleneck of traditional DML. The resulting models are not only more accurate but also more efficient, providing a scalable framework for high-performance metric learning across various domains.
Summary updated