Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax

Andres Potapczynski, Gabriel Loaiza-Ganem, John P. Cunningham

Abstract

The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments.

Introduction

Discrete variables are ubiquitous in machine learning, appearing in latent variable models for language, reinforcement learning, and structured prediction. However, the non-differentiable nature of discrete sampling prevents the use of standard backpropagation. Continuous relaxations, such as the Gumbel-Softmax (GS), have become the standard solution. Despite its popularity, GS suffers from several limitations: it is restricted to Gumbel noise, lacks a closed-form KL divergence for the relaxation itself, and cannot easily handle distributions with countably infinite support.

This paper introduces Invertible Gaussian Reparameterization (IGR), a framework that replaces Gumbel noise with Gaussian noise and utilizes invertible transformations to map samples onto the simplex. This approach yields a more flexible family of distributions with significant theoretical and empirical advantages.

Methods

The IGR framework generates samples by first drawing standard Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$, which is transformed into a latent vector $y = \mu + \sigma\epsilon$. This vector is then mapped to the simplex via an invertible function $g(y, \tau)$, where $\tau$ is a temperature hyperparameter.

Two primary mapping functions are explored:

  1. Softmax++: A modified, invertible version of the softmax function that ensures the output remains within the valid simplex bounds.
  2. Stick-Breaking (SB): A procedure that allows the model to represent an infinite number of categories by iteratively partitioning a unit-length interval.

A major theoretical benefit of IGR is that the invertibility of $g$ allows for a closed-form calculation of the KL divergence between two IGR distributions. It is simply the KL divergence between the underlying Gaussian distributions, which is computationally efficient and avoids the variance associated with Monte Carlo approximations.

Results and Evaluation

The authors evaluated IGR against Gumbel-Softmax using standard benchmarks including MNIST, Fashion-MNIST (FMNIST), and Omniglot.

Approximating Infinite Support

A key strength of IGR-SB is its ability to handle distributions with infinite support, such as the Poisson distribution. As illustrated in the following comparison, IGR-SB accurately captures the target distribution's shape and tail. In contrast, Gumbel-Softmax requires a fixed, large number of categories ($K$) and still fails to match the distribution's tail effectively.

Comparison of IGR-SB and Gumbel-Softmax (GS) in approximating a target distribution p. IGR-SB (top) converges to the target more accurately than GS (bottom), which struggles even with high K values.

Quantitative Performance in VAEs

In Variational Autoencoder (VAE) experiments, IGR variants consistently outperformed GS. As shown in the table below, IGR-I (using Softmax++) and IGR-SB achieved higher test log-likelihoods across all three datasets, demonstrating the empirical robustness of the proposed framework.

Quantitative results for VAEs on MNIST, FMNIST, and Omniglot. IGR variants (IGR-I, IGR-SB) consistently outperform the Gumbel-Softmax (GS) baseline.

Discussion and Conclusion

IGR demonstrates that sacrificing the direct interpretability of latent parameters in favor of mathematical flexibility leads to superior performance. By leveraging invertible transformations and Gaussian noise, IGR provides a theoretically sound alternative to Gumbel-Softmax with closed-form KL divergence and the ability to handle infinite categories. Future research may further explore the integration of normalizing flows within the IGR framework to enhance flexibility in complex Bayesian modeling tasks.

Summary updated

Abstract

The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments. Our code is available at https://github.com/cunningham-lab/igr.
BibTeX
@inproceedings{potapczynski2020invertible,
  author = {Andres Potapczynski and Gabriel Loaiza-Ganem and John P. Cunningham},
  title = {{Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=CRVan7Qd0u},
}
Figure