Dataset Dynamics via Gradient Flows in Probability Space

David Alvarez-Melis, Nicolò Fusi

Introduction

In traditional machine learning, the standard paradigm involves keeping a dataset fixed while optimizing a model's parameters to minimize error. When data is manipulated—such as through augmentation, sub-sampling, or "mixup" techniques—it is typically done using heuristics rather than a unified mathematical theory. This paper introduces a principled framework for transforming labeled datasets by treating them as probability distributions and moving them along "gradient flows." By optimizing the data itself rather than just the model, the authors demonstrate how to "re-purpose" frozen classifiers for entirely new tasks and improve transfer learning through geometric data alignment.

The Problem: Fixed Data, Flexible Models

While there are established methods to transform unlabeled data (such as generative modeling), doing so for labeled datasets—where features $x$ and labels $y$ are tightly coupled—is significantly harder. Existing domain adaptation methods often assume the label sets between domains are identical or rely on specific, inflexible model architectures.

This paper seeks a model-independent, geometrically grounded way to "shape" labeled datasets. The goal is to manipulate the data to satisfy specific constraints or to seamlessly match a target domain, shifting the focus from "how do I change my model to fit this data?" to "how do I change my data to fit this model?"

Methodology: Flowing Data in Probability Space

The authors propose viewing a dataset as a collection of samples from a joint probability distribution $\rho(x, y)$. They then cast dataset transformation as an optimization problem in the space of probability distributions: $\min_{\rho} F(\rho)$.

To solve this, they utilize Wasserstein Gradient Flows. Imagine the dataset as a cloud of particles in a high-dimensional space. A "gradient flow" defines a mathematical path that these particles follow to minimize a functional $F$.

Gradient flows of label distributions over time, illustrating how classes move and interact.

Key components of this framework include:

  • Optimal Transport Dataset Distance (OTDD): A metric used to measure how "far apart" two labeled datasets are, even if they have completely different label sets (e.g., comparing a dataset of digits to one of letters). By representing labels as distributions over features, Optimal Transport provides a purely geometric way to align data.
  • Functional Objectives: The authors define three types of "energies" to steer the data:
  • Potential Energy: Enforces per-instance constraints (e.g., making classes linearly separable).
  • Interaction Energy: Models how samples relate to each other (e.g., pushing different classes apart).
  • Wasserstein Distance ($T_\beta$): Pulls the dataset toward a target "reference" dataset.
  • Particle Dynamics: They derive three ways to implement the flow:
  • Feature-driven (fd): Only the features $x$ move.
  • Joint-driven fixed-label (jd-fl): Features and label distributions move, but a point's original label identity stays the same.
  • Joint-driven variable-label (jd-vl): Points can effectively change their labels as they move through the space.

Key Results: Repurposing Models and Shaping Data

The framework was evaluated on several standard image classification benchmarks (MNIST, USPS, Fashion-MNIST, KMNIST, CIFAR-10, STL-10) and CAMELYON, a histopathology dataset used to test flows between highly dissimilar domains.

One of the most striking applications of this framework is Model Re-purposing. The authors demonstrate that you can take a "frozen" black-box model trained on Dataset A and use it to classify Dataset B with high accuracy, simply by "flowing" Dataset B into the domain of Dataset A.

  • Zero-Shot Re-purposing: A frozen model trained only on MNIST (digits) achieved an astounding 99.1% accuracy on Fashion-MNIST and 99.2% on KMNIST after the target data was flowed to the MNIST domain using a supervised flow.
  • Unsupervised Success: Even without using target labels (relying on pseudo-labels instead), the MNIST model reached ~60% accuracy on FMNIST/KMNIST, massively outperforming the ~5-8% baseline.
  • Transfer Learning: In 100-shot scenarios, using the "trajectories" of the flow as additional training data consistently outperformed standard fine-tuning and other data augmentation baselines.

Furthermore, the authors successfully demonstrated Geometric Shaping. They were able to shape synthetic datasets to become linearly separable or to follow complex geometries, such as a "Swiss Roll," while maintaining their original class identities.

Transforming an initial dataset to match a reference "Swiss Roll" geometry using different functional objectives over time.

Limitations and Future Directions

While mathematically elegant and highly effective, the framework has limitations. Optimal Transport is computationally expensive, especially for large datasets. Although the authors use Gaussian approximations to speed up label distances, scaling to massive datasets remains a challenge. Additionally, mapping continuous "flowed" label distributions back to discrete categories requires clustering, which can be sensitive to hyperparameters. Finally, many of the most powerful applications require a reference dataset to "pull" the flow toward.

Future work may explore integrating deep learning tools within this framework, such as using neural networks to parameterize the flows (similar to normalizing flows) or applying these geometric data techniques to meta-learning.

Summary updated

Abstract

Various machine learning tasks, from generative modeling to domain adaptation, revolve around the concept of dataset transformation and manipulation. While various methods exist for transforming unlabeled datasets, principled methods to do so for labeled (e.g., classification) datasets are missing. In this work, we propose a novel framework for dataset transformation, which we cast as optimization over data-generating joint probability distributions. We approach this class of problems through Wasserstein gradient flows in probability space, and derive practical and efficient particle-based methods for a flexible but well-behaved class of objective functions. Through various experiments, we show that this framework can be used to impose constraints on classification datasets, adapt them for transfer learning, or to re-purpose fixed or black-box models to classify -- with high accuracy -- previously unseen datasets.
BibTeX
@inproceedings{alvarezmelis2021dataset,
  author = {David Alvarez-Melis and Nicolò Fusi},
  title = {{Dataset Dynamics via Gradient Flows in Probability Space}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=0wA0KhPXqag},
}
Figure