Generative modeling has seen a massive leap forward with techniques like GANs and Diffusion Models. However, training these models often relies on measuring the "distance" between the empirical data distribution and the data the model generates. While the Wasserstein distance (Optimal Transport) is a gold standard for this, it is notoriously expensive to compute in high dimensions.
A new paper from researchers at UT Austin, titled "Amortized Projection Optimization for Sliced Wasserstein Generative Models," introduces a clever way to make these computations faster and more effective by "learning to optimize" the distance metric itself.
TL;DR
The authors propose Amortized Sliced Wasserstein (A-SW), a method that replaces expensive iterative searches for optimal projection directions with a fast neural network "predictor." This eliminates the slow nested optimization loops in current models, leading to faster training and better image generation quality on standard benchmarks.
What Problem Does This Paper Address?
To train a generative model, you need a loss function that tells the model how far its generated images are from real ones. The Sliced Wasserstein (SW) distance is popular because it breaks down high-dimensional data into one-dimensional "slices" (projections), which are much easier and faster to compare.
The challenge lies in obtaining a good distance measure. You either need thousands of random projections (which is memory-intensive) or you need to find the best projection that maximizes the distance (Max-SW). Finding this "max" projection requires an iterative optimization loop inside every single step of your main model training. This "nested loop" makes training painfully slow and computationally expensive.
Methods
The researchers bridge the gap between Optimal Transport and Amortized Optimization. Instead of running a mini-optimization every time they need a projection, they train a separate, lightweight "amortized model" ($f_\psi$) to predict the best projection direction directly from the data.
They introduce three flavors of these amortized models:
- Linear (LA-SW): Predicts directions based on a linear combination of the input samples.
- Generalized Linear (GA-SW): Adds non-linear activations (like Sigmoids) to the input before predicting.
- Non-linear (NA-SW): Uses a more complex neural network structure to map inputs to the optimal projection.
By using these models, the training process turns into a minimax game (similar to a GAN): the generator tries to minimize the distance, while the amortized model tries to find the most "informative" projection to maximize it.
Key Insights and Ideas
- Breaking the Nested Loop: The core insight is realizing that we don't need the perfect projection at every step; we just need a model that gets better at finding them as the generator improves.
- Lower Bounding Max-SW: The paper proves that A-SW acts as a lower bound to the Max-SW distance. This provides a theoretical safety net for using a predicted projection instead of an optimized one.
- Efficiency vs. Expressiveness: Interestingly, the simplest model (LA-SW) often performed the best. It strikes a perfect balance: it's fast enough not to slow down training but expressive enough to find much better directions than random sampling.
- Memory Savings: By predicting a single "good" direction instead of sampling 10,000 random ones, the model significantly reduces the memory footprint during training.
Key Results
The authors evaluated their approach on four standard computer vision benchmarks (CIFAR10, STL10, CelebA, and CelebA-HQ) and compared A-SW against standard SNGANs, Sliced Wasserstein (SW), and Max-Sliced Wasserstein (Max-SW).
- Better Quality: On CIFAR10, LA-SW achieved an FID score (lower is better) of 13.21, outperforming standard SW (14.25) and Max-SW (31.33).
- Faster Convergence: As shown in the training curves above, A-SW models reached high Inception Scores (IS) and low FID scores much earlier in the training process than the baselines across all evaluated datasets.
- Speed and Memory: LA-SW is nearly as fast as standard SW with only one projection but delivers results comparable to (or better than) SW with 1,000+ projections. For example, on CelebA-HQ, LA-SW processed ~10.19 iterations/sec, while Max-SW with 100 iterations slowed down to ~8.13 iterations/sec.
Limitations and Future Work
While A-SW is powerful, the authors note a few areas for growth:
- Not a True Metric: Technically, A-SW doesn't satisfy the "identity of indiscernibles" (it might not be exactly zero even if two distributions are identical), though this doesn't seem to hurt generative performance in practice.
- Architecture Sensitivity: The performance of A-SW depends on the choice of the amortized model. While they tested three types, there might be even better architectures (like Convolutions or Transformers) for specific data types.
- Scaling Up: Future work could explore using A-SW in even higher-resolution settings or applying it to non-image data like point clouds or graphs.
For those interested in the implementation, the authors have released their code on GitHub.
Summary updated