TL;DR
This paper tackles the challenge of applying optimal transport (OT) to infinite-dimensional spaces, like those needed for functional data. They introduce a novel Gaussian smoothing technique to overcome "spurious solutions" that plague existing neural OT methods in these complex settings, leading to more accurate and stable learning of transport maps.
What Problem Does This Paper Address?
Optimal Transport (OT) is a powerful framework for matching probability distributions. While successful in low dimensions, scaling OT to high-dimensional data, particularly functional data (like time series or functions), presents significant challenges. Neural OT methods aim to learn these transport maps using neural networks. However, in "non-regular" settings – where data distributions aren't "smooth" enough – these methods often produce "spurious solutions." These are incorrect transport maps that satisfy the optimization objective but don't accurately represent the true underlying data transformation. This problem is particularly acute in infinite-dimensional Hilbert spaces, where the concept of "regularity" (like absolute continuity with respect to Lebesgue measure in finite dimensions) is not directly applicable. The paper aims to provide a theoretically sound and practically effective way to overcome these spurious solutions in infinite-dimensional OT.
Methods
The core of the paper's contribution lies in two main areas:
- Theoretical Characterization of Spurious Solutions: The authors extend the concept of "regular measures" to infinite-dimensional Hilbert spaces. This provides a theoretical foundation for understanding when OT problems are well-posed and when spurious solutions might arise. They analytically characterize the conditions under which the semi-dual neural OT (SNOT) formulation is well-posed and recovers a unique Monge map.
- Gaussian Smoothing Strategy: To address the ill-posedness in non-regular settings, they propose a regularization strategy based on Gaussian smoothing. This involves convolving the source measure with a Gaussian measure. The key insight is that the geometry of this Gaussian smoothing (defined by its covariance operator) is crucial. They prove a necessary and sufficient condition for this smoothing to make the problem well-posed, relating it to the projection of the source measure onto the kernel of the covariance operator. This strategy is then integrated into the SNOT framework, creating "Hilbert Semi-dual Neural Optimal Transport" (HiSNOT). They also propose an "annealed" smoothing schedule, gradually reducing the noise during training to transition from a regularized problem to the original, potentially singular, one.
Datasets
The paper evaluates their HiSNOT model on:
- Synthetic functional data: Specifically designed scenarios (Perpendicular, Horizontal, One-to-Many, Multi-Perpendicular) to test the model's ability to handle different types of singular distributions and spurious solution issues.
- Real-world time-series imputation benchmarks: Including datasets like ETTh1, ETTh2, ETTm1, ETTm2, and Exchange. These are treated as functional data problems where imputation is framed as an optimal transport task.
Key Insights and Ideas
- Infinite-dimensional OT is tricky: The paper clearly demonstrates that standard OT assumptions break down in infinite-dimensional spaces, leading to fundamental issues like spurious solutions.
- "Regularity" is key, and Gaussian smoothing can create it: The concept of "regular measures" is a crucial generalization of absolute continuity for infinite dimensions. Gaussian smoothing is shown to be a principled way to achieve this regularity.
- The geometry of smoothing matters: It's not just about adding noise; how you add it (defined by the covariance operator) is critical. The noise needs to align with the "singular directions" of the data to effectively regularize the problem.
- Spurious solutions are a real problem for neural OT: The examples provided vividly illustrate how SNOT can fail without proper regularization, producing maps that don't respect the underlying data distributions.
- Annealing is a practical way to train: Gradually reducing the Gaussian noise during training allows the model to first learn a well-posed problem and then refine its solution towards the original, potentially singular, target.
Key Results
- Theoretical guarantees: The paper provides theoretical guarantees for the well-posedness of SNOT in Hilbert spaces under regular source measures and proves that Gaussian smoothing can restore this regularity.
- Suppression of spurious solutions: Empirical results on synthetic data show that HiSNOT with Gaussian smoothing significantly outperforms the vanilla SNOT model in scenarios prone to spurious solutions. For instance, in the "Perpendicular" task, the transport cost error (Dcost) was reduced from 0.101 to 0.038, and the target distribution error (Dtarget) from 0.249 to 0.006.
- State-of-the-art performance on time-series imputation: HiSNOT achieves competitive or state-of-the-art results on several time-series imputation benchmarks, demonstrating its practical utility for real-world functional data problems. Notably, on the Exchange dataset, it achieved an MSE of 0.004, an order of magnitude lower than the next best performer.
Limitations and Future Work
- Convergence guarantee up to a subsequence: The paper acknowledges that their convergence guarantee (Theorem 4.2) holds up to a subsequence. This is attributed to the potential non-uniqueness of optimal transport plans in general, meaning the training process might oscillate between valid solutions. However, they assure that any limit point obtained is a valid optimal transport plan.
- Theoretical extension of finite-dimensional conditions: The authors note that a direct infinite-dimensional analogue of the finite-dimensional condition for avoiding spurious solutions (related to Hausdorff dimension) remains an open question.
- Further exploration of Gaussian null sets: While the paper defines Gaussian null sets and their role in regularity, further theoretical exploration of their properties in infinite dimensions could be beneficial.
Summary updated