TL;DR
This paper introduces Spherical Sliced Fused Gromov Wasserstein (SSFG), a novel method for measuring the distance between probability distributions in the latent space of autoencoders. By replacing uniform "slicing" directions with learnable, concentrated distributions (such as the von Mises-Fisher distribution), the authors create more effective relational regularizers. This targeted approach significantly improves both image generation and reconstruction quality in autoencoder models.
What Problem Does This Paper Address?
Autoencoders (AEs) are foundational deep learning models used to compress data into a lower-dimensional "latent space" and subsequently reconstruct it. To enable these models to generate new data, researchers often regularize the latent space so that it adheres to a specific prior distribution, like a Mixture of Gaussians.
A state-of-the-art approach for this is the Relational Regularized Autoencoder (RAE), which uses a technique called "slicing" to compare distributions. Slicing projects high-dimensional data onto 1D lines where distances are computationally cheaper to evaluate. However, standard RAEs sample these 1D projection directions uniformly from a sphere.
The underlying problem is that in high-dimensional spaces, most random directions are "non-informative"—they fail to capture the meaningful differences between the distributions. This uniform sampling leads to an underestimation of the true discrepancy, resulting in a poorly structured latent space that ultimately degrades the model's ability to generate sharp, realistic images.
Methods
To overcome the limitations of uniform sampling, the authors propose a family of new discrepancies that intelligently search for the most informative projection directions:
- SSFG (Spherical Sliced Fused Gromov Wasserstein): Instead of uniform sampling, SSFG utilizes the von Mises-Fisher (vMF) distribution. The vMF distribution features a "mode" (a preferred direction) and a "concentration" parameter. This allows the model to focus its slices on the most critical areas of the latent space.
- MSSFG (Mixture SSFG): Recognizing that complex data might require multiple optimal projection directions, this variant employs a mixture of vMFs to capture several clusters of important directions simultaneously.
- PSSFG (Power SSFG): Sampling vMFs in very high dimensions can become a computational bottleneck. To address this, the authors introduce the Power Spherical distribution, which serves as a faster, more numerically stable alternative.
These new discrepancies are integrated into the RAE framework, resulting in three novel model variants: s-DRAE, ms-DRAE, and ps-DRAE.
Key Insights and Ideas
- Interpolation is Key: The authors demonstrate that SSFG acts as a bridge between standard uniform slicing and "max-slicing" (which exclusively evaluates the single best direction). By tuning the concentration parameter, the model finds an optimal balance that captures more structural information than either extreme.
- Beating the Curse of Dimensionality: The paper provides theoretical guarantees (Theorem 3) showing that SSFG is statistically efficient and does not suffer from the "curse of dimensionality" during inference, making it highly robust for deep learning applications.
- Learnable Projections: By making the parameters of the slicing distribution learnable, the autoencoder essentially "learns how to look" at its own latent space, dynamically identifying where it deviates most from the desired prior.
Key Results
The proposed models were evaluated on standard generative modeling benchmarks, including MNIST and CelebA, and consistently outperformed the previous state-of-the-art (DRAE) as well as other popular models like VAE and WAE.
- Generative Quality (FID Score): On the MNIST dataset, the ms-DRAE achieved an impressive FID score of 43.57, marking a significant improvement over the standard DRAE baseline of 58.04. Lower FID scores indicate more realistic and higher-quality generated images.
- Reconstruction: The new models also demonstrated superior reconstruction capabilities. On MNIST, ms-DRAE reached a reconstruction loss of 11.12, compared to 14.07 for the baseline DRAE.
- Speed vs. Quality: While vMF-based models (s-DRAE) introduce slight training overhead due to complex sampling, the ps-DRAE (Power Spherical) variant strikes an excellent balance, delivering high-quality results with significantly lower computational demands.
Limitations and Future Work
- Hyperparameter Tuning: The success of these models relies on selecting the appropriate concentration ($\\kappa$) and the number of mixture components ($K$). While the authors provide recommended search ranges, this introduces an additional step to the training pipeline.
- Computational Overhead: Even with the optimized "Power" variant, these methods remain more computationally intensive than a standard VAE due to the optimization steps required to find the best slicing directions.
- Broader Applications: The authors note that these spherical sliced discrepancies hold promise for other domains, such as domain adaptation, clustering, and improving GAN stability, presenting exciting avenues for future research.
Summary updated