Unsupervised Domain Adaptation for Distance Metric Learning

Kihyuk Sohn, Wenling Shang, Xiang Yu, Manmohan ChandrakerICLR 2019 · OpenReview

This paper tackles a critical flaw in how we train AI to recognize faces across different ethnicities, introducing a novel framework for Unsupervised Domain Adaptation (UDA) specifically designed for distance metric learning.

TL;DR

Most domain adaptation techniques fail when the source and target datasets contain entirely different categories (e.g., different people). This paper introduces the Feature Transfer Network (FTN), which reformulates classification as a verification task to bridge disjoint label spaces. This significantly improves face recognition accuracy for underrepresented ethnicities without requiring new labels.

The Problem: Disjoint Label Spaces and Demographic Bias

Deep learning models for face recognition often suffer from "demographic bias." Because popular datasets like MS-Celeb-1M are predominantly Caucasian, models perform poorly on African-American or East-Asian faces.

Standard UDA techniques usually try to solve domain shifts by making the features of the "source" (Caucasian) and "target" (non-Caucasian) domains indistinguishable. However, there is a logical trap here: in face recognition, different people should have different features. If the source domain contains Person A and the target domain contains Person B, forcing their feature distributions to overlap actually destroys the model's ability to tell individuals apart. The authors identify that we need a way to transfer knowledge about how to distinguish faces without forcing the actual identities to merge in the embedding space.

Methods: Bridging the Gap

The authors propose three key innovations to solve the disjoint label problem:

  1. Verification Reformulation: Instead of treating the problem as a multi-class classification task (which requires shared labels), they treat it as a binary verification task: "Are these two images the same person?" This task is universal across all domains, regardless of which specific identities are present.
  2. Feature Transfer Network (FTN):
FTN Architecture

The architecture uses a shared feature generator ($f$) but adds a "Feature Transfer" module ($g$). While the generator keeps source and target features separate (to maintain identity), the transfer module aligns a transformed version of the source space with the target space. This allows the model to learn the "style" of the target domain while keeping the "content" (identities) distinct.

  1. Multi-class Entropy Minimization (MCEM): To boost performance on the unlabeled target domain, the authors use an offline clustering algorithm (HDBSCAN) to discover "pseudo-identities." They then apply a powerful metric learning loss ($N$-pair loss) to these clusters, encouraging the model to push different pseudo-identities apart and pull the same ones together.

Datasets and Key Insights

The researchers evaluated their approach on both a controlled proof-of-concept and real-world face recognition datasets.

  • MNIST-M to MNIST (Proof of Concept): A toy setup where the source domain contains digits 0–4 and the target domain contains digits 5–9. This tests the model's ability to adapt when classes are entirely disjoint.
  • Cross-Ethnicity Faces (CEF): A dataset derived from MS-Celeb-1M, adapting from Caucasian (CAU) source identities to African-American (AA) and East-Asian (EA) target identities.

Key Results

  • Visual Clarity:
t-SNE visualization of MNIST-M to MNIST adaptation

In the MNIST experiment, t-SNE visualizations showed that standard adversarial adaptation (DANN) caused significant confusion between digits (e.g., mixing 3s and 5s). In contrast, the proposed FTN maintained 10 clean, separate clusters, proving that alignment should happen in a transformed space rather than the primary embedding space.

  • Face Recognition Gains: On the CEF dataset, the FTN+MCEM model significantly outperformed the "source-only" baseline. For example, identification accuracy on African-American faces jumped from 69.64% (baseline) to 80.75% (FTN+MCEM).
  • Closing the Gap: The proposed method's performance closely approached the "upper bound" (a model trained with full labels for all ethnicities), proving that unsupervised adaptation can be nearly as effective as manual labeling.

Limitations and Future Work

  • Clustering Dependency: The success of the MCEM module depends on the quality of the initial clustering. If the feature space is initially very poor, HDBSCAN might produce noisy pseudo-labels that hinder learning.
  • Computational Overhead: The process involves offline clustering and complex $N$-pair sampling, which is more computationally intensive than standard classification-based adaptation.
  • Generalization: While highly effective for faces, the paper doesn't explore how this framework handles other metric learning tasks, such as fine-grained image retrieval (e.g., identifying specific car models or bird species).

Summary updated

Abstract

Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.
BibTeX
@inproceedings{sohn2019unsupervised,
  author = {Kihyuk Sohn and Wenling Shang and Xiang Yu and Manmohan Chandraker},
  title = {{Unsupervised Domain Adaptation for Distance Metric Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=SYnMBbjoBVGA},
}
Figure