Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein

Khai Nguyen, Son Nguyen, Nhat Ho, Tung Pham, Hung Bui

Abstract

Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regularization, we propose a new relational discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that can find an important area of projections characterized by a von Mises-Fisher distribution. Then, we introduce two variants of SSFG to improve its performance. The first variant, named mixture spherical sliced fused Gromov Wasserstein (MSSFG), replaces the vMF distribution by a mixture of von Mises-Fisher distributions to capture multiple important areas of directions that are far from each other. The second variant, named power spherical sliced fused Gromov Wasserstein (PSSFG), replaces the vMF distribution by a power spherical distribution to improve the sampling time in high dimension settings. We then apply the new discrepancies to the RAE framework to achieve its new variants. Finally, we conduct extensive experiments to show that the new proposed autoencoders have favorable performance in learning latent manifold structure, image generation, and reconstruction.

Introduction

Relational Regularized Autoencoders (RAEs) aim to map complex data into a structured low-dimensional latent space. A critical challenge in this framework is ensuring that the distribution of encoded data matches a target prior distribution. While Sliced Fused Gromov-Wasserstein (SFG) has been used to measure this discrepancy by taking 1D projections (slices) of the distributions, it typically relies on uniform sampling of these directions. In high-dimensional spaces, uniform slicing is often inefficient because many random directions fail to capture the meaningful structural differences between distributions.

Methodology: Spherical Sliced Fused Gromov Wasserstein

To overcome the limitations of uniform slicing, we propose Spherical Sliced Fused Gromov Wasserstein (SSFG). Instead of treating all directions equally, SSFG learns an informative distribution of slicing directions using the von Mises-Fisher (vMF) distribution. The vMF distribution acts as a spherical analogue to the Gaussian distribution, defined by a location parameter (the center of the projection area) and a concentration parameter $\kappa$.

Comparison of uniform slicing versus the proposed von Mises-Fisher distribution for selecting informative directions.

By optimizing these parameters, the model can focus its "slicing" on the most discriminative regions of the latent space. We extend this core idea into two specialized variants:

  • Mixture SSFG (MSSFG): Uses a mixture of vMF distributions to simultaneously capture multiple important clusters of directions.
  • Power Spherical SSFG (PSSFG): Utilizes a Power Spherical distribution, which offers superior computational efficiency and numerical stability in very high-dimensional settings compared to standard vMF.
The Relational Regularized Autoencoder framework incorporating Spherical Sliced Fused Gromov Wasserstein for latent space regularization.

These discrepancies are integrated into the RAE framework to create new autoencoder variants: s-DRAE, ms-DRAE, and ps-DRAE.

Experiments and Results

We evaluated our proposed models on standard benchmarks, including MNIST and CelebA, focusing on image generation and latent manifold structure.

Key Findings

  • Generative Quality: The mixture variant (ms-DRAE) significantly outperformed standard DRAE and other baselines like VAE and Wasserstein Autoencoder (WAE). On MNIST, ms-DRAE achieved a Frechet Inception Distance (FID) of 43.57, compared to 58.04 for the standard DRAE. On CelebA, it achieved a leading FID of 46.01.
  • Structural Stability: Qualitative results showed that spherical regularization produces clearer, more recognizable images and effectively prevents "mode collapse," a common issue where models fail to represent the full diversity of the data.
  • Efficiency: While learning the slicing distribution adds some overhead, the ps-DRAE variant provides a high-performance middle ground, maintaining better FID scores than random slicing while remaining computationally viable for high-dimensional data.

Conclusion

By replacing uniform random projections with learned spherical distributions, SSFG provides a more targeted and effective regularization for autoencoders. This approach allows models to focus on the most informative geometric features of the latent space, resulting in superior generative performance and more robust manifold learning.

Summary updated

Abstract

Relational regularized autoencoder (RAE) is a framework to learn the distribution of data by minimizing a reconstruction loss together with a relational regularization on the latent space. A recent attempt to reduce the inner discrepancy between the prior and aggregated posterior distributions is to incorporate sliced fused Gromov-Wasserstein (SFG) between these distributions. That approach has a weakness since it treats every slicing direction similarly, meanwhile several directions are not useful for the discriminative task. To improve the discrepancy and consequently the relational regularization, we propose a new relational discrepancy, named spherical sliced fused Gromov Wasserstein (SSFG), that can find an important area of projections characterized by a von Mises-Fisher distribution. Then, we introduce two variants of SSFG to improve its performance. The first variant, named mixture spherical sliced fused Gromov Wasserstein (MSSFG), replaces the vMF distribution by a mixture of von Mises-Fisher distributions to capture multiple important areas of directions that are far from each other. The second variant, named power spherical sliced fused Gromov Wasserstein (PSSFG), replaces the vMF distribution by a power spherical distribution to improve the sampling time in high dimension settings. We then apply the new discrepancies to the RAE framework to achieve its new variants. Finally, we conduct extensive experiments to show that the new proposed autoencoders have favorable performance in learning latent manifold structure, image generation, and reconstruction.
BibTeX
@inproceedings{nguyen2021improving,
  author = {Khai Nguyen and Son Nguyen and Nhat Ho and Tung Pham and Hung Bui},
  title = {{Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=ljfdnoRbLsJ},
}
Figure