Symmetrical Synthesis for Deep Metric Learning

Geonmo Gu, ByungSoo Ko

TL;DR

This paper introduces "symmetrical synthesis," a novel, hyperparameter-free method for generating synthetic hard negative samples in deep metric learning. By creating symmetric points around existing feature pairs, it effectively augments training data, leading to improved performance in clustering and retrieval tasks without increasing model complexity or training time.

What Problem Does This Paper Address?

Deep metric learning aims to learn embeddings where similar items are close and dissimilar items are far apart. A crucial aspect of training these models is selecting "hard" samples – those that are difficult for the model to distinguish correctly. Existing methods for generating synthetic hard samples, like those using autoencoders or GANs, often introduce significant complexity, more hyperparameters, and slower training. This paper addresses the need for a simpler, more efficient, and plug-and-play method to generate these informative synthetic samples, thereby improving the effectiveness of deep metric learning without these drawbacks.

Methods

The proposed method, "symmetrical synthesis," operates in two main steps:

  1. Symmetrical Synthetic Generation: Given two original feature points ($x_i$, $x_j$) from the same class in the embedding space, the method generates new synthetic points ($x_i'$, $x_j'$) using the line connecting $x_i$ and $x_j$ as an axis of symmetry. This is achieved through simple algebraic computation, making it hyperparameter-free. The paper notes that parameters $\alpha$ and $\beta$ are used for explanation and experimentation but are not actual hyperparameters for the core method.
  2. Hard Negative Pair Mining: After generating these synthetic points, the method combines them with the original points to form a larger pool of potential negative samples. It then performs hard negative pair mining within this augmented set to select the most informative negative pairs for computing the metric learning loss. This strategy aims to push dissimilar classes further apart more effectively.

The method is designed to be plug-and-play, meaning it can be integrated with existing metric learning losses (like triplet loss, N-pair loss, angular loss, and lifted structure loss) without requiring any modifications to the network architecture.

Datasets

The paper evaluates its method on three widely used benchmarks:

  • CUB-200-2011 (CUB200): A dataset of bird images.
  • CARS196: A dataset of car images.
  • Standard Online Products (SOP): A large-scale dataset of product images.

The standard train/test splits for these datasets are used, with specific numbers of images and classes detailed for each. The evaluation is performed without bounding box information for CUB200 and CARS196.

Key Insights and Ideas

  • Simplicity is Key: The core insight is that complex generative models (GANs, autoencoders) are not necessary for generating effective synthetic hard samples. Simple geometric transformations can achieve similar or better results with significantly less complexity.
  • Symmetry as a Constraint: Using symmetry as the generation principle ensures that the synthetic points maintain a consistent relationship (cosine similarity and Euclidean distance) with their original counterparts. This property is crucial for not disrupting the positive pair relationships in metric learning losses.
  • Augmenting the Search Space: By creating synthetic points, the method effectively expands the pool of potential hard negatives. This allows metric learning losses to exploit more informative samples that might otherwise be missed by traditional mining strategies.
  • Hyperparameter-Free Advantage: The method's lack of hyperparameters is a significant practical advantage, simplifying its integration and reducing the burden of tuning.
  • Impact of Norm and Similarity: Experiments show that maintaining the correct similarity and norm of synthetic points is critical for stable training and good performance. Deviations can lead to untrainable models or performance degradation.

Key Results

The proposed symmetrical synthesis method consistently improves performance across various metric learning losses and datasets.

  • Clustering and Retrieval Boost: Across CUB200, CARS196, and SOP datasets, the method, when combined with different metric learning losses (Triplet, N-pair, Angular, Lifted-Struct), shows significant gains in both clustering (NMI, F1 scores) and retrieval (R@1, R@2, R@4, R@8, R@10, R@100) metrics compared to baselines and other state-of-the-art methods like DAML and HDML.
  • Competitive Performance: In most cases, the proposed method achieves the best scores, outperforming existing hard sample generation techniques. For instance, on the SOP dataset, it provides a 5.2 to 11.8% performance boost in Recall@1, demonstrating its effectiveness even on large datasets.
  • Efficiency: The computational cost and memory consumption of symmetrical synthesis are negligible, adding only a small overhead to the training process. The paper reports a marginal increase in training time per batch.

Limitations and Future Work

  • Limitations:
    • The paper notes that in one specific comparison on the CARS196 dataset with N-pair loss, HDML (N-pair) slightly outperformed Symm + N-pair. This is speculated to be due to using the same hyperparameters as HDML for fair comparison, implying that further tuning of Symm + N-pair might yield even better results.
  • The paper mentions that synthetic points might not always be perfectly positioned within the "ideal" cluster boundary at the very beginning of training, leading to initial fluctuations.
  • Future Work:
    • The paper suggests that the method's effectiveness is demonstrated across various losses and datasets, implying its broad applicability. Further exploration could involve combining it with other advanced metric learning techniques or applying it to different domains beyond image retrieval and clustering.
  • The visualization of synthetic points suggests they are useful as hard negatives even if not perfectly within a cluster, hinting at potential for further analysis of their placement and impact.

Summary updated

Abstract

Deep metric learning aims to learn embeddings that contain semantic similarity information among data points. To learn better embeddings, methods to generate synthetic hard samples have been proposed. Existing methods of synthetic hard sample generation are adopting autoencoders or generative adversarial networks, but this leads to more hyper-parameters, harder optimization, and slower training speed. In this paper, we address these problems by proposing a novel method of synthetic hard sample generation called symmetrical synthesis. Given two original feature points from the same class, the proposed method firstly generates synthetic points with each other as an axis of symmetry. Secondly, it performs hard negative pair mining within the original and synthetic points to select a more informative negative pair for computing the metric learning loss. Our proposed method is hyper-parameter free and plug-and-play for existing metric learning losses without network modification. We demonstrate the superiority of our proposed method over existing methods for a variety of loss functions on clustering and image retrieval tasks. Our implementations is publicly available.
BibTeX
@inproceedings{gu2020symmetrical,
  author = {Geonmo Gu and ByungSoo Ko},
  title = {{Symmetrical Synthesis for Deep Metric Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6716},
}
Figure