Metric Learning in Optimal Transport for Domain Adaptation

Tanguy Kerdoncuff, Rémi Emonet, Marc Sebban

Abstract

Domain Adaptation (DA) remains a significant challenge in machine learning, where models trained on source data often fail on target data due to distribution shifts. This paper introduces MLOT (Metric Learning in Optimal Transport), a novel framework that improves unsupervised domain adaptation by jointly learning a custom Mahalanobis distance metric and a transportation plan. A key theoretical contribution is the proof that Principal Component Analysis (PCA) is the optimal linear projection for minimizing Wasserstein distances, providing a rigorous foundation for the method. MLOT consistently outperforms standard Optimal Transport techniques on computer vision benchmarks.

Introduction

The performance of machine learning models is frequently compromised when the deployment environment (target domain) differs from the training environment (source domain). Unsupervised Domain Adaptation (UDA) seeks to mitigate this "domain shift" without requiring labels for the target data. Optimal Transport (OT) has emerged as a powerful tool for UDA, treating data distributions as masses to be moved efficiently. However, standard OT relies on fixed metrics like Euclidean distance, which may not align with the intrinsic geometry of the data shift.

Methodology: Metric Learning in Optimal Transport

The proposed MLOT algorithm addresses the limitations of fixed-metric OT by jointly optimizing dimensionality reduction, the transportation plan, and the ground metric.

PCA as a Wasserstein Optimizer

A significant theoretical insight of this work is the connection between classical dimensionality reduction and modern transport theory. The authors prove that PCA is the optimal linear projection for minimizing the Wasserstein distance between a high-dimensional distribution and its lower-dimensional representation. This result justifies the use of PCA as a starting point for domain alignment.

PCA minimizes the Wasserstein distance between the original distribution and its projection.

The MLOT Algorithm

MLOT operates through an iterative process that alternates between two primary steps:

  1. Optimal Transport: Given the current metric, the Sinkhorn-Knopp algorithm is used to find the most efficient transportation plan between domains.
  2. Metric Learning: The source domain's metric is updated via gradient descent to minimize alignment error, guided by a new generalization bound derived by the authors.
Overview of the MLOT process, integrating PCA and Optimal Transport.

Experimental Evaluation

The authors evaluated MLOT on standard computer vision benchmarks, including Office-Caltech and Office31, using both traditional SURF features and modern DeCAF deep learning features.

Key Results

  • SURF Features: On the Office-Caltech dataset, MLOT achieved an average accuracy of 49.7%, significantly outperforming standard OTDA (48.2%) and the "No Adaptation" baseline (31.4%).
  • Deep Features: Utilizing DeCAF6 features, MLOT reached 84.7% accuracy, representing a 1.5-point improvement over the next best non-deep-learning method.
  • Robustness: The method demonstrated superior performance in difficult cross-domain scenarios, such as adapting from Webcam images to Amazon product images.

Discussion and Conclusion

MLOT demonstrates that learning the ground metric is essential for effective Optimal Transport in domain adaptation. By integrating metric learning with dimensionality reduction and transportation planning, the method provides a theoretically grounded and empirically successful approach to UDA. While currently focused on linear Mahalanobis distances, future work may extend this framework to non-linear shifts and end-to-end deep learning architectures.

Summary updated

Abstract

Domain Adaptation aims at benefiting from a labeled dataset drawn from a source distribution to learn a model from examples generated from a different but related target distribution. Creating a domain-invariant representation between the two source and target domains is the most widely technique used. A simple and robust way to perform this task consists in (i) representing the two domains by subspaces described by their respective eigenvectors and (ii) seeking a mapping function which aligns them. In this paper, we propose to use Optimal Transport (OT) and its associated Wassertein distance to perform this alignment. While the idea of using OT in domain adaptation is not new, the original contribution of this paper is two-fold: (i) we derive a generalization bound on the target error involving several Wassertein distances. This prompts us to optimize the ground metric of OT to reduce the target risk; (ii) from this theoretical analysis, we design an algorithm (MLOT) which optimizes a Mahalanobis distance leading to a transportation plan that adapts better. Extensive experiments demonstrate the effectiveness of this original approach.
BibTeX
@inproceedings{kerdoncuff2020metric,
  author = {Tanguy Kerdoncuff and Rémi Emonet and Marc Sebban},
  title = {{Metric Learning in Optimal Transport for Domain Adaptation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/299},
}
Figure