A Unified Framework for Domain Adaptation Using Metric Learning on Manifolds

Sridhar Mahadevan, Bamdev Mishra, Shalini GhoshECML/PKDD 2018 · OpenReview

TL;DR

This paper introduces a novel framework for domain adaptation that unifies geometric and statistical differences between datasets by leveraging the Riemannian geometry of covariance matrices. By formulating domain adaptation as a "geometric mean metric learning" problem, the authors achieve state-of-the-art results on computer vision benchmarks, demonstrating improved performance over existing methods.

What Problem Does This Paper Address?

Machine learning models often require large amounts of labeled data, which can be expensive and time-consuming to acquire. Domain adaptation (DA) aims to address this by transferring knowledge from a labeled "source" domain to an unlabeled "target" domain, where the data distributions might differ. Traditional DA methods often struggle to reconcile both the statistical differences (e.g., changes in feature distributions) and the underlying geometric structures of these domains. This paper tackles the challenge of creating a unified approach that can effectively handle both types of discrepancies.

Methods

The core of the proposed method lies in treating covariance matrices as points on a Riemannian manifold. Specifically, the authors exploit the geometry of Symmetric Positive Definite (SPD) matrices, which form a curved manifold. They formulate domain adaptation as a problem of learning a Mahalanobis distance matrix, which is achieved by minimizing an objective function that involves the geometric mean of source and target covariance matrices. This geometric mean computation is elegantly reduced to solving a Riccati equation.

The framework is extended in several ways:

  • GCA1 (Geometric Mean Metric Learning): Aligns source and target covariance matrices using their geometric mean.
  • GCA2 (MMD Integration): Incorporates the Maximum Mean Discrepancy (MMD) metric to further reduce statistical differences between domains.
  • GCA3 (Manifold Geometry Integration): Adds constraints based on the underlying manifold geometry of the source and target domains, modeled using diffusion on graphs.
  • Cascaded Variants: These methods apply the geometric mean computation in a nested fashion, allowing for more complex interactions between different constraints.

A key advantage highlighted is the uniqueness and theoretical optimality of their solutions, derived from the properties of the Riccati equation and the convex nature of their objective functions on the SPD manifold.

Datasets

The authors evaluate their methods on standard computer vision benchmark datasets:

  • Office: A dataset with 31 categories.
  • Office-Caltech10: An extended version of the Office dataset with 10 object categories.

These datasets are commonly used in transfer learning and domain adaptation research, allowing for direct comparison with prior work.

Key Insights and Ideas

  • Covariance Matrices as Manifold Points: Viewing covariance matrices as points on a Riemannian manifold is a powerful abstraction that allows for principled geometric operations (like computing geodesics and geometric means) that are not possible in standard Euclidean space.
  • Riccati Equation as a Unifying Tool: The reduction of complex domain adaptation objectives to solving Riccati equations provides a mathematically elegant and computationally efficient way to integrate various constraints.
  • Unified Framework for Diverse Constraints: The ability to combine geometric alignment, statistical discrepancy minimization (MMD), and manifold diffusion within a single framework is a significant advancement.
  • Cascaded Geometric Means for Deeper Alignment: The cascaded approach suggests that multiple layers of geometric mean computations can lead to more nuanced and effective alignment between domains.

Key Results

The proposed methods (GCA1, GCA2, GCA3, and their cascaded variants) consistently outperform previous domain adaptation techniques, including CORAL, TCA, and GFK, on the Office and Office-Caltech10 datasets. Table 1 shows that the proposed methods achieve higher recognition accuracy in a majority of the 12 transfer learning tasks. For instance, in the Amazon to DSLR task, the cascaded GCA3 method shows significant improvements. The paper also demonstrates that the performance of the GCA methods is sensitive to hyperparameter choices (t, γ, μ), but that optimal settings lead to substantial gains.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of the proposed methods is influenced by hyperparameter choices, as illustrated in Figures 2, 4, and 5. The paper suggests that finding optimal hyperparameters is crucial.
  • Theoretical Analysis of Variance: While experimental results show performance improvements, a deeper theoretical analysis of why the performance varies across different transfer tasks is noted as a future direction.
  • Nonlinear Variants and Deep Learning: The authors suggest extending their approach to nonlinear variants and developing deep learning versions where gradients can tune neural network weights. This could potentially lead to improved feature learning and even better performance.
  • Exploration of Other Metric Learning Methods: The paper proposes that their framework can be used to design new domain adaptation methods by leveraging other metric learning techniques.

Summary updated

Abstract

We present a novel framework for domain adaptation, whereby both geometric and statistical differences between a labeled source domain and unlabeled target domain can be reconciled using a unified mathematical framework that exploits the curved Riemannian geometry of statistical manifolds. We exploit a simple but important observation that as the space of covariance matrices is both a Riemannian space as well as a homogeneous space, the shortest path geodesic between two covariances on the manifold can be computed analytically. Statistics on the SPD matrix manifold, such as the geometric mean of two SPD matries can be reduced to solving the well-known Riccati equation. We show how the Ricatti-based solution can be constrained to not only reduce the statistical differences between the source and target domains, such as aligning second order covariances and minimizing the maximum mean discrepancy, but also the underlying geometry of the source and target domains using diffusions on the underlying source and target manifolds. Our solution also emerges as a consequence of optimal transport theory, which shows that the optimal transport mapping between source and target distributions that are multivariate Gaussians is a function of the geometric mean of the source and target covariances, a quantity that also minimizes the Wasserstein distance. A key strength of our proposed approach is that it enables integrating multiple sources of variation between source and target in a unified way, by reducing the combined objective function to a nested set of Ricatti equations where the solution can be represented by a cascaded series of geometric mean computations. In addition to showing the theoretical optimality of our solution, we present detailed experiments using standard transfer learning testbeds from computer vision comparing our proposed algorithms to past work in domain adaptation, showing improved results over a large variety of previous methods. Code related to this paper is available at: https://github.com/sridharmahadevan/Geodesic-Covariance-Alignment .
BibTeX
@inproceedings{mahadevan2018unified,
  author = {Sridhar Mahadevan and Bamdev Mishra and Shalini Ghosh},
  title = {{A Unified Framework for Domain Adaptation Using Metric Learning on Manifolds}},
  booktitle = {ECML/PKDD},
  year = {2018},
  url = {https://openreview.net/forum?id=a0H_EqLwxM3},
}
Figure