Abstract
The Wasserstein distance is a powerful tool for comparing probability distributions, yet its computational complexity and lack of differentiability often limit its application in deep learning. We present a novel approach using permutation-invariant neural networks to embed sample sets into a low-dimensional Euclidean space where the Euclidean distance serves as a high-fidelity approximation of the Wasserstein metric. By utilizing a Siamese DeepSets architecture and incorporating geometric regularizers, our model achieves high correlation ($r > 0.95$) with ground-truth Sinkhorn distances and demonstrates robust generalization to unseen distribution types and discrete measures.
Introduction
In modern machine learning, comparing probability distributions is a fundamental task. The Wasserstein distance, derived from Optimal Transport theory, is highly regarded for its ability to respect the underlying geometry of the metric space. However, it faces significant practical hurdles: it scales cubically with sample size, is not Hadamard differentiable, and is sensitive to outliers. While entropic regularization (Sinkhorn distance) offers some speedup, it remains computationally demanding for real-time applications.
This research explores whether a neural network can learn a direct mapping from the infinite-dimensional space of probability measures into a simple, low-dimensional Euclidean space. The goal is to create an embedding where the standard Euclidean distance accurately reflects the Wasserstein distance, effectively "flattening" the complex geometry of distribution space.
Methodology
The authors treat the problem as a set-encoding task, utilizing a Siamese network architecture with a DeepSets backbone. DeepSets is essential for this application because it is permutation-invariant; the representation of a distribution must remain identical regardless of the order in which its samples are processed.
Training and Regularization
The network is trained by encoding two sets of samples into low-dimensional vectors and minimizing the difference between their Euclidean distance and the calculated Sinkhorn distance. To ensure the learned space adheres to the mathematical properties of the Wasserstein metric, the authors introduced custom regularizers:
- Translation Invariance: Shifting distributions equally in the input space should not change their relative distance in the embedding space.
- Scaling Law: Scaling the distributions should result in a proportional scaling of the distance.
Results and Key Insights
The model demonstrates remarkable accuracy and geometric intuition. A key finding is that the network naturally learns statistical moments without explicit supervision. For 1D distributions mapped to a 2D embedding, the x-coordinate correlates strongly with the distribution's mean, while the y-coordinate correlates with its variance.
Performance and Generalization
As shown in Figure 2, the network achieves near-perfect correlation ($r \approx 0.98$ to $1.00$) across various scenarios, including in-training distributions, out-of-training densities, and 2D distributions. Notably, the model exhibits zero-shot generalization, accurately computing distances for discrete distributions (like Dirac and Binomial measures) despite being trained only on continuous ones.
Furthermore, the latent space preserves the geometry of transformations. When a distribution is translated along a circular path, its embedding also traces a circle. The space also supports the approximation of Wasserstein barycenters; the geometric average of two distributions can be found by simply identifying the Euclidean midpoint between their embeddings.
Discussion and Limitations
While the model excels at approximating the 1-Wasserstein ($W_1$) metric, it faces challenges with the 2-Wasserstein ($W_2$) metric. This is likely due to the non-negative sectional curvature of $W_2$ space, which makes it mathematically difficult to embed into a flat Euclidean space without distortion. Additionally, the model's performance is sensitive to sample size, requiring at least 100-500 samples per set to maintain consistency.
Future work aims to mathematically prove the continuity properties of these networks and explore higher-dimensional embeddings to capture more complex statistical features like skewness and kurtosis.
Summary updated