Abstract
The notion of task similarity is at the core of various machine learning paradigms, such as domain adaptation and meta-learning. Current methods to quantify it are often heuristic, make strong assumptions on the label sets across the tasks, and many are architecture-dependent, relying on task-specific optimal parameters. In this work, we propose the Optimal Transport Dataset Distance (OTDD), an alternative notion of distance between datasets that is model-agnostic, does not involve training, can compare datasets even if their label sets are completely disjoint, and has a solid theoretical footing. Our results show that this novel distance provides meaningful comparison of datasets and correlates well with transfer learning hardness across various experimental settings.
Introduction
In machine learning, quantifying the similarity between two datasets is crucial for predicting the success of transfer learning, domain adaptation, and meta-learning. However, datasets consist of both features and labels, making comparison difficult—especially when label sets do not overlap. Existing solutions often rely on "probes," which involve training a model on one dataset and evaluating it on another. This process is computationally expensive and introduces bias based on the specific model architecture used. This paper introduces a learning-free, mathematically rigorous distance metric that leverages the geometry of the data.
Methodology: Optimal Transport Dataset Distance (OTDD)
The core innovation of OTDD is treating a dataset as a joint distribution of features and labels and using Optimal Transport (OT) to compare these distributions.
Labels as Distributions
Instead of treating labels as arbitrary integers, OTDD represents each label as the distribution of features associated with it. For example, the label "dog" is represented by the "cloud" of all dog images in the feature space. This allows the distance between two labels to be calculated as the Wasserstein distance between their respective feature distributions.
The Ground Metric and Computational Efficiency
To compare two data points $(x, y)$ and $(x', y')$, the authors define a distance that combines feature-space distance (e.g., Euclidean) and label-space distance. Computing the exact Wasserstein distance between every pair of labels is computationally intensive ($O(n^5 \log n)$). To make this practical, the authors approximate each label's feature distribution as a Gaussian, allowing for a closed-form solution that drastically speeds up the calculation.
Key Insights and Results
OTDD provides a geometry-aware distance that naturally handles disjoint labels. Because labels are defined by their features, the method can discover that "MNIST 1s" are closer to "USPS 1s" than to "MNIST 8s" without prior knowledge of the digit relationships.
Predicting Transferability
A key finding is that this geometric distance correlates strongly with actual model performance. If Dataset A is geometrically close to Dataset B, a model trained on A will likely transfer well to B. In experiments across the *NIST datasets, OTDD showed a strong negative correlation (Pearson $\rho = -0.59$) with the relative drop in test error during transfer. In NLP tasks using BERT embeddings, the correlation remained significant ($\rho = -0.59$).
Informing Data Augmentation
OTDD can also quantitatively evaluate data augmentations. For instance, it can show that certain rotations make MNIST look "further" from USPS, explaining why those specific augmentations might degrade transfer performance.
Conclusion and Future Work
OTDD offers a principled, efficient way to measure dataset similarity without the need for model training. While currently requiring datasets to share the same feature dimensionality, future work may incorporate Gromov-Wasserstein distances to compare datasets with entirely different feature types. Despite the Gaussian assumption used for speed, the metric remains robust and provides a valuable "pre-flight" check for transfer learning tasks.
Summary updated