Overview
In machine learning, we often need to answer a fundamental question: "How similar is Dataset A to Dataset B?" This is crucial for transfer learning (choosing the best pre-training data) and domain adaptation.
However, quantifying "dataset distance" is surprisingly hard. While features are easy to compare (e.g., Euclidean distance between pixels), labels are tricky. How do you measure the distance between a "dog" label in one dataset and a "truck" label in another? Most existing solutions are either heuristic, require training a computationally expensive "probe" model on both datasets, or only work if both datasets share the exact same classes.
To solve this, researchers introduce OTDD (Optimal Transport Dataset Distance), a principled, model-agnostic way to measure the distance between two datasets. OTDD doesn't require training a model and can compare datasets even if their labels are completely different, providing a metric that highly correlates with how well knowledge will transfer from one task to another.
Methods: Optimal Transport for Datasets
The researchers turn to Optimal Transport (OT)—a mathematical framework for moving "mass" from one distribution to another with minimal cost.
The Core Innovation: Hierarchical OT
The authors treat a dataset as a collection of feature-label pairs $(x, y)$. To find the distance between two datasets, they solve a two-level problem:
- Label Distance: They represent each label (e.g., "Category: 3") as a distribution of the features associated with it. They then use the Wasserstein distance to calculate how far apart these label-induced distributions are. This allows them to compare "apples" to "oranges" by looking at the geometry of the data points that define them.
- Dataset Distance: Once they have a distance between labels, they combine it with the distance between features to create a "ground cost." They then solve a global OT problem to find the minimum cost to "transform" one entire dataset into the other.
Scaling Up: Bures-OTDD
Exact OT is computationally heavy ($O(n^3 \log n)$). To make this practical for large-scale ML, the authors propose Bures-OTDD. By approximating the label distributions as Gaussians, they can use an analytical solution (the Bures-Wasserstein distance) for the inner loop, drastically speeding up the calculation while maintaining a tight bound on the true distance.
Key Insights and Ideas
- Labels are Distributions: The "aha moment" is realizing that a label isn't just a name; it's a signature in the feature space. By treating labels as distributions, the metric can discover that a "1" in the MNIST dataset is "close" to a "1" in USPS without any prior knowledge of their semantics.
- Geometry over Heuristics: Because OT is "geometry-aware," it captures the underlying structure of the data. As illustrated below, label-agnostic distances (like feature-only OT) fail to capture the difficulty of a task when label distributions shift, whereas OTDD successfully captures this divergence.
- Predicting Transferability: The most important insight is that this geometric distance is a proxy for "transfer hardness." If two datasets are close in OTDD space, a model trained on one will likely perform well (or fine-tune quickly) on the other.
Key Results
The authors validated their metric across several modalities, including vision (MNIST, CIFAR-10, Tiny-ImageNet) and text (sentence classification using BERT embeddings).
- High Correlation with Transfer Success: OTDD showed a strong negative correlation (Pearson $\rho$ between $-0.59$ and $-0.85$) with transfer learning success. As the OTDD distance between a source and target dataset decreased, the "relative drop in error" (the gain from pre-training) increased significantly.
- Efficiency: The Bures-OTDD approximation proved to be remarkably tight. For datasets with thousands of samples, the Gaussian approximation becomes a highly accurate and viable option.
- Robustness to Augmentations: The metric correctly identified that cropping images is "closer" to the original task than rotating them by 180 degrees, matching empirical observations in computer vision.
Limitations and Future Work
- Same-Dimension Features: Currently, the method assumes features are in the same space (or can be embedded into one, like using BERT for text). If Dataset A has 100 features and Dataset B has 500, you can't directly use this specific formulation. The authors suggest using Gromov-Wasserstein distances to solve this in the future.
- Gaussian Assumption: The fast Bures-OTDD assumes classes are normally distributed. While this works well for many embeddings, it might struggle with highly complex, multi-modal classes (e.g., a "furniture" class containing both lamps and sofas).
- Ignoring the Model: While being model-agnostic is a strength, sometimes the specific architecture (e.g., a ResNet vs. a Transformer) changes how "distant" two tasks feel. Integrating model capacity into the metric is a suggested next step.
Summary updated