Abstract
The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments.
Introduction
Discrete variables are ubiquitous in machine learning, appearing in latent variable models for language, reinforcement learning, and structured prediction. However, the non-differentiable nature of discrete sampling prevents the use of standard backpropagation. Continuous relaxations, such as the Gumbel-Softmax (GS), have become the standard solution. Despite its popularity, GS suffers from several limitations: it is restricted to Gumbel noise, lacks a closed-form KL divergence for the relaxation itself, and cannot easily handle distributions with countably infinite support.
This paper introduces Invertible Gaussian Reparameterization (IGR), a framework that replaces Gumbel noise with Gaussian noise and utilizes invertible transformations to map samples onto the simplex. This approach yields a more flexible family of distributions with significant theoretical and empirical advantages.
Methods
The IGR framework generates samples by first drawing standard Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$, which is transformed into a latent vector $y = \mu + \sigma\epsilon$. This vector is then mapped to the simplex via an invertible function $g(y, \tau)$, where $\tau$ is a temperature hyperparameter.
Two primary mapping functions are explored:
- Softmax++: A modified, invertible version of the softmax function that ensures the output remains within the valid simplex bounds.
- Stick-Breaking (SB): A procedure that allows the model to represent an infinite number of categories by iteratively partitioning a unit-length interval.
A major theoretical benefit of IGR is that the invertibility of $g$ allows for a closed-form calculation of the KL divergence between two IGR distributions. It is simply the KL divergence between the underlying Gaussian distributions, which is computationally efficient and avoids the variance associated with Monte Carlo approximations.
Results and Evaluation
The authors evaluated IGR against Gumbel-Softmax using standard benchmarks including MNIST, Fashion-MNIST (FMNIST), and Omniglot.
Approximating Infinite Support
A key strength of IGR-SB is its ability to handle distributions with infinite support, such as the Poisson distribution. As illustrated in the following comparison, IGR-SB accurately captures the target distribution's shape and tail. In contrast, Gumbel-Softmax requires a fixed, large number of categories ($K$) and still fails to match the distribution's tail effectively.
Quantitative Performance in VAEs
In Variational Autoencoder (VAE) experiments, IGR variants consistently outperformed GS. As shown in the table below, IGR-I (using Softmax++) and IGR-SB achieved higher test log-likelihoods across all three datasets, demonstrating the empirical robustness of the proposed framework.
Discussion and Conclusion
IGR demonstrates that sacrificing the direct interpretability of latent parameters in favor of mathematical flexibility leads to superior performance. By leveraging invertible transformations and Gaussian noise, IGR provides a theoretically sound alternative to Gumbel-Softmax with closed-form KL divergence and the ability to handle infinite categories. Future research may further explore the integration of normalizing flows within the IGR framework to enhance flexibility in complex Bayesian modeling tasks.
Summary updated