Abstract
Deep metric learning (DML) is a cornerstone of many computer vision applications. It aims at learning a mapping from the input domain to an embedding space, where semantically similar objects are located nearby and dissimilar objects far from another. The target similarity on the training data is defined by user in form of ground-truth class labels. However, while the embedding space learns to mimic the user-provided similarity on the training data, it should also generalize to novel categories not seen during training. Besides user-provided groundtruth training labels, a lot of additional visual factors (such as viewpoint changes or shape peculiarities) exist and imply different notions of similarity between objects, affecting the generalization on the images unseen during training. However, existing approaches usually directly learn a single embedding space on all available training data, struggling to encode all different types of relationships, and do not generalize well. We propose to build a more expressive representation by jointly splitting the embedding space and the data hierarchically into smaller sub-parts. We successively focus on smaller subsets of the training data, reducing its variance and learning a different embedding subspace for each data subset. Moreover, the subspaces are learned jointly to cover not only the intricacies, but the breadth of the data as well. Only after that, we build the final embedding from the subspaces in the conquering stage. The proposed algorithm acts as a transparent wrapper that can be placed around arbitrary existing DML methods. Our approach significantly improves upon the state-of-the-art on image retrieval, clustering, and re-identification tasks evaluated using CUB200-2011, CARS196, Stanford Online Products, In-shop Clothes, and PKU VehicleID datasets.
Introduction
Deep Metric Learning (DML) is the fundamental process of training neural networks to map images into a multi-dimensional space where semantically similar items are clustered together and dissimilar items are pushed apart. This technology powers critical applications like image search, face recognition, and product retrieval.
A primary hurdle in DML is intra-class variability. For instance, two images of a specific car model might appear drastically different due to lighting or viewpoint, while two different models might look nearly identical from a specific angle. Traditional DML approaches attempt to learn a single, global embedding space for all data simultaneously. This often results in "mode collapse," where the model ignores subtle visual nuances to satisfy broad class labels, or overfits to training categories, failing to generalize to unseen objects.
The Divide and Conquer Framework
To address these challenges, this paper introduces a hierarchical Divide and Conquer strategy. This framework acts as a transparent wrapper that can be integrated with existing DML loss functions, such as Triplet or Margin loss.
1. Divide: Data and Embedding Space
The algorithm begins by training on the entire dataset. Periodically, it employs K-means clustering within the current embedding space to partition the data into smaller, more homogeneous subsets. This reduction in variance within each sub-problem allows the model to focus on fine-grained, local details.
Simultaneously, the embedding space is divided into dedicated subspaces. Rather than increasing the total number of dimensions, the framework uses "masks"—weight vectors that isolate specific dimensions of the shared feature representation for each data cluster.
2. Progressive Training and Conquering
Training proceeds recursively, acting as a curriculum. The model might start with a single cluster and progressively split into 2, 4, 8, and up to 32 sub-problems. This transition from broad similarities to intricate relationships prevents the model from taking "shortcuts" (like relying solely on background color).
In the final Conquer stage, the individual subspace masks are merged into a single, unified mapping. The network then undergoes a brief fine-tuning phase to ensure these specialized subspaces harmonize into a robust global representation.
Key Insights: Why It Works
Implicit Hard Mining and Manifold Efficiency
By clustering the data, the model naturally groups "hard negatives"—different classes that look visually similar—into the same sub-problem. This makes training more efficient without requiring expensive global search strategies. Furthermore, visualization techniques show that this approach utilizes the embedding space more uniformly, avoiding the high-density "hubs" that typically signal overfitting in baseline models.
Capturing Diverse Visual Factors
A key advantage of the subspace approach is its ability to capture different notions of similarity. As shown in retrieval results, different subspaces may focus on distinct attributes like color, shape, or specific parts of an object, which are then combined for a more comprehensive final embedding.
Experimental Results
The proposed method was evaluated on five standard DML benchmarks, including CUB200-2011 (birds), CARS196, and Stanford Online Products (SOP). It consistently outperformed state-of-the-art techniques:
- PKU VehicleID: Improved Recall@1 by over 4.2% compared to the baseline.
- Stanford Online Products: Achieved a Recall@1 of 80.32%, a significant margin over previous best results.
- Parameter Efficiency: The approach shares the majority of parameters across sub-problems, achieving superior results to massive ensembles while using only a fraction of the parameters (26.6M vs. 537M).
Conclusion
The Divide and Conquer framework provides a powerful, scalable way to improve the expressiveness and generalization of Deep Metric Learning. By breaking down the complex task of global similarity into manageable, homogeneous sub-problems, it forces the model to learn robust, intrinsic features that translate effectively to novel, unseen categories.
Summary updated