Geometric Dataset Distances via Optimal Transport

David Alvarez-Melis, Nicoló Fusi

Abstract

The notion of task similarity is at the core of various machine learning paradigms, such as domain adaptation and meta-learning. Current methods to quantify it are often heuristic, make strong assumptions on the label sets across the tasks, and many are architecture-dependent, relying on task-specific optimal parameters. In this work, we propose the Optimal Transport Dataset Distance (OTDD), an alternative notion of distance between datasets that is model-agnostic, does not involve training, can compare datasets even if their label sets are completely disjoint, and has a solid theoretical footing. Our results show that this novel distance provides meaningful comparison of datasets and correlates well with transfer learning hardness across various experimental settings.

Introduction

In machine learning, quantifying the similarity between two datasets is crucial for predicting the success of transfer learning, domain adaptation, and meta-learning. However, datasets consist of both features and labels, making comparison difficult—especially when label sets do not overlap. Existing solutions often rely on "probes," which involve training a model on one dataset and evaluating it on another. This process is computationally expensive and introduces bias based on the specific model architecture used. This paper introduces a learning-free, mathematically rigorous distance metric that leverages the geometry of the data.

Methodology: Optimal Transport Dataset Distance (OTDD)

The core innovation of OTDD is treating a dataset as a joint distribution of features and labels and using Optimal Transport (OT) to compare these distributions.

Labels as Distributions

Instead of treating labels as arbitrary integers, OTDD represents each label as the distribution of features associated with it. For example, the label "dog" is represented by the "cloud" of all dog images in the feature space. This allows the distance between two labels to be calculated as the Wasserstein distance between their respective feature distributions.

The Ground Metric and Computational Efficiency

To compare two data points $(x, y)$ and $(x', y')$, the authors define a distance that combines feature-space distance (e.g., Euclidean) and label-space distance. Computing the exact Wasserstein distance between every pair of labels is computationally intensive ($O(n^5 \log n)$). To make this practical, the authors approximate each label's feature distribution as a Gaussian, allowing for a closed-form solution that drastically speeds up the calculation.

Figure 2: Comparison of feature-only OT vs. OTDD. OTDD correctly identifies dataset similarity by accounting for label distributions, even when feature-only methods fail.

Key Insights and Results

OTDD provides a geometry-aware distance that naturally handles disjoint labels. Because labels are defined by their features, the method can discover that "MNIST 1s" are closer to "USPS 1s" than to "MNIST 8s" without prior knowledge of the digit relationships.

Predicting Transferability

A key finding is that this geometric distance correlates strongly with actual model performance. If Dataset A is geometrically close to Dataset B, a model trained on A will likely transfer well to B. In experiments across the *NIST datasets, OTDD showed a strong negative correlation (Pearson $\rho = -0.59$) with the relative drop in test error during transfer. In NLP tasks using BERT embeddings, the correlation remained significant ($\rho = -0.59$).

Figure 4: Visualization of OTDD components: label-to-label distances (left), optimal coupling between samples in feature space (center), and the resulting transport plan (right).

Informing Data Augmentation

OTDD can also quantitatively evaluate data augmentations. For instance, it can show that certain rotations make MNIST look "further" from USPS, explaining why those specific augmentations might degrade transfer performance.

Conclusion and Future Work

OTDD offers a principled, efficient way to measure dataset similarity without the need for model training. While currently requiring datasets to share the same feature dimensionality, future work may incorporate Gromov-Wasserstein distances to compare datasets with entirely different feature types. Despite the Gaussian assumption used for speed, the metric remains robust and provides a valuable "pre-flight" check for transfer learning tasks.

Summary updated

Abstract

The notion of task similarity is at the core of various machine learning paradigms, such as domain adaptation and meta-learning. Current methods to quantify it are often heuristic, make strong assumptions on the label sets across the tasks, and many are architecture-dependent, relying on task-specific optimal parameters (e.g., require training a model on each dataset). In this work we propose an alternative notion of distance between datasets that (i) is model-agnostic, (ii) does not involve training, (iii) can compare datasets even if their label sets are completely disjoint and (iv) has solid theoretical footing. This distance relies on optimal transport, which provides it with rich geometry awareness, interpretable correspondences and well-understood properties. Our results show that this novel distance provides meaningful comparison of datasets, and correlates well with transfer learning hardness across various experimental settings and datasets.
BibTeX
@inproceedings{alvarezmelis2020geometric,
  author = {David Alvarez-Melis and Nicoló Fusi},
  title = {{Geometric Dataset Distances via Optimal Transport}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=i01orKvUB49},
}

Loading related content...

Figure