Training Triplet Networks with GAN

Maciej Zieba, Lei Wang

TL;DR

This paper introduces a novel framework that trains a Triplet Network by embedding it as the discriminator within a Generative Adversarial Network (GAN). By combining supervised triplet-based metric learning with unsupervised generative training, the model achieves significantly better feature representations, especially when labeled data is scarce.

What Problem Does This Paper Address?

In computer vision, we often want to learn a "metric"—a way to represent images such that similar images are close together and different images are far apart. Triplet Networks are a popular tool for this; they learn by comparing a "query" image to a "positive" (similar) and "negative" (dissimilar) example. However, these networks typically require a large volume of labeled triplets to perform well.

On the other hand, GANs are excellent at learning feature representations from unlabeled data by forcing a discriminator to distinguish between real and "fake" generated images. The authors address the gap between these two worlds: How can we use the unsupervised power of GANs to boost the performance of triplet-based metric learning?

Methods

The authors propose a hybrid architecture where the Triplet Network $T(x)$ serves as the backbone of the GAN's discriminator.

  1. The Discriminator ($D$): Instead of a standard binary classifier, the discriminator's probability of an image being "real" is derived directly from the $M$ features produced by the triplet network.
  2. The Generator ($G$): The generator is trained using a feature matching technique. Rather than just trying to "fool" the discriminator, the generator tries to produce images that result in internal feature activations similar to those of real images.
  3. The Loss Function: The model uses a dual-purpose loss:
  4. Supervised Part: A log-probability triplet loss that encourages the network to correctly rank distances between query, positive, and negative samples.
  5. Unsupervised Part: A standard GAN loss where the discriminator learns to tell real data from generated data, and the generator learns to mimic the data distribution.

By training these simultaneously, the "triplet" features are enriched by the GAN's attempt to understand the underlying structure of the data, even from unlabeled samples.

Datasets

The approach was evaluated on two classic benchmark datasets:

  • MNIST: Handwritten digits (60,000 training, 10,000 test). The authors tested with as few as 100 labeled examples.
  • Cifar10: Natural images across 10 classes (50,000 training, 10,000 test). The authors used 5,000 labeled examples.

Key Insights and Ideas

  • Feature Enrichment: The "aha moment" is that the GAN training acts as a powerful regularizer. Even if you have very few labeled triplets, the process of generating fake images and trying to distinguish them forces the network to learn more robust, meaningful features than triplet training alone.
  • Efficiency in Low Dimensions: The model performs remarkably well even when the output feature vector is very small (e.g., only 16 dimensions). This is highly desirable for fast $k$-nearest neighbor ($k$-nn) searches in large databases.
  • Synergy: The GAN provides the "global" context of what the data looks like, while the Triplet loss provides the "local" context of how specific classes should be separated.

Key Results

The hybrid method significantly outperformed baselines that used only Triplet loss or only GAN-based feature extraction.

Classification accuracy comparison on MNIST and CIFAR-10.
Note: Results are based on 9-nn evaluation using only 16 features.

On Cifar10, the improvement was particularly striking, showing a ~10% jump over the standard triplet network. The authors also achieved a Mean Average Precision (mAP) of 0.6525 on Cifar10, outperforming other semi-supervised GAN variants.

Limitations and Future Work

  • The High-Dimension Paradox: Interestingly, the authors observed that performance sometimes decreased when the number of features increased beyond 128.
Performance degradation as feature dimensions (m) increase.

They attribute this to the "hard mining" technique (selecting the most difficult triplets), which can lead to overfitting on specific hard cases at the expense of general accuracy when the feature space is too large.

  • Static vs. Dynamic Mining: Currently, hard triplets are selected once per epoch. The authors suggest that moving to a dynamic, per-minibatch update for triplet selection could solve the overfitting issues in higher dimensions.
  • Advanced GANs: The authors noted that incorporating more modern GAN variants, such as Wasserstein GAN (WGAN), could further stabilize training and improve results.

Summary updated

Abstract

Triplet networks are widely used models that are characterized by good performance in classification and retrieval tasks. In this work we propose to train a triplet network by putting it as the discriminator in Generative Adversarial Nets (GANs). We make use of the good capability of representation learning of the discriminator to increase the predictive quality of the model. We evaluated our approach on Cifar10 and MNIST datasets and observed significant improvement on the classification performance using the simple k-nn method.
BibTeX
@inproceedings{zieba2017training,
  author = {Maciej Zieba and Lei Wang},
  title = {{Training Triplet Networks with GAN}},
  booktitle = {Workshops @ ICLR},
  year = {2017},
  url = {https://openreview.net/forum?id=ELWBsqfSeD},
}
Figure