Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax

Andres Potapczynski, Gabriel Loaiza-Ganem, John P. Cunningham

Introduction

Invertible Gaussian Reparameterization (IGR) is a flexible, modular alternative to the Gumbel-Softmax for relaxing discrete distributions. By swapping Gumbel noise for Gaussian noise and using invertible transformations, IGR achieves closed-form KL divergence, handles countably infinite categories, and consistently outperforms the Gumbel-Softmax in variational autoencoders and nonparametric modeling.

Background: The Differentiability Problem

Many machine learning tasks—from reinforcement learning to generative modeling—rely on discrete stochastic components. However, the discrete nature of these components makes them "non-differentiable," preventing the use of standard backpropagation.

To solve this, researchers use the reparameterization trick, which transforms independent noise into a sample from the desired distribution. For discrete variables, the gold standard has been the Gumbel-Softmax (GS). While GS is widely used, it has several rigidities:

  1. Fixed Architecture: It requires specific Gumbel noise and the softmax function.
  2. No Closed-Form KL: The KL divergence between two GS distributions usually requires noisy Monte Carlo estimates.
  3. Finite Support: It struggles to scale to distributions with an infinite number of categories (e.g., Poisson or geometric distributions).
  4. Interpretability Trade-off: GS parameters are easily interpreted as discrete probabilities, but the authors argue this "mathematically pleasing" quality might actually be holding back model performance.

Proposed Method: Invertible Gaussian Reparameterization

The authors introduce Invertible Gaussian Reparameterization (IGR). Instead of Gumbel noise, IGR starts with standard Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$. This noise is shifted and scaled ($y = \mu + \sigma \epsilon$) and then passed through an invertible function $g(y, \tau)$ that maps the values onto the simplex (the space of probability distributions).

Key technical innovations include:

  • Softmax++: A modified, invertible version of the softmax function that maps onto the open simplex.
  • Stick-Breaking IGR: A mapping based on the stick-breaking process, which allows the model to handle an infinite number of categories ($K = \infty$) by dynamically "truncating" the distribution based on data.
  • Normalizing Flows: Because the framework is modular, the authors show you can even use normalizing flows as the mapping function to increase flexibility.
  • Closed-Form KL: Because the mapping is invertible, the KL divergence between two IGR distributions is simply the KL divergence between their underlying Gaussians, which is calculated analytically. This removes a significant source of noise during training.

Key Insights and Ideas

  • Giving up Interpretability for Performance: The "aha moment" of the paper is that we don't actually need the parameters of our continuous relaxation to look like discrete probabilities. By treating the parameters as abstract latent variables and using a small "moment matching" step to recover discrete probabilities later, we gain massive flexibility.
  • Modularity is King: Unlike the Gumbel-Softmax, which is a single "take it or leave it" distribution, IGR is a framework. You can swap out the mapping function (Softmax++, Stick-breaking, or Flows) depending on whether you need speed, infinite categories, or maximum expressivity.
  • Stability through Math: By achieving a closed-form KL divergence, IGR provides a much more stable gradient signal than the Gumbel-Softmax, which relies on Monte Carlo sampling for its KL term.

Experimental Results

The authors evaluated IGR across several standard benchmarks and synthetic tasks, including MNIST, FMNIST, Omniglot, and a Poisson distribution approximation task.

Superior VAE Performance

In Variational Autoencoder (VAE) experiments on MNIST, FMNIST, and Omniglot, IGR variants (IGR-I and IGR-SB) consistently achieved higher test log-likelihoods than the Gumbel-Softmax.

VAE Performance Table

Infinite Support

IGR-SB successfully approximated a Poisson distribution ($\lambda=50$) without needing to pre-specify the number of categories $K$. The Gumbel-Softmax requires a fixed $K$, and its performance degrades if $K$ is chosen poorly.

Approximating Infinite Support

Nonparametric Modeling and Efficiency

IGR enabled truly nonparametric inference in Dirichlet Process models, outperforming existing methods while avoiding expensive hyperparameter searches for the number of components. Furthermore, while the "Planar Flow" version of IGR is about 30% slower due to its complexity, the standard IGR-I and IGR-SB variants have execution times almost identical to the Gumbel-Softmax.

Limitations and Future Work

  • Parameter Recovery: Because IGR parameters aren't directly interpretable, you must perform a simple numerical integration or Monte Carlo step to recover the actual discrete probabilities (though the authors show this cost is negligible).
  • Complexity of Flows: Using complex normalizing flows within IGR increases computational overhead and prevents the use of the closed-form KL divergence.
  • Beyond the Simplex: While this paper focuses on categorical distributions, future work could extend the invertible Gaussian approach to other discrete structures like permutations or graphs.

Summary updated

Abstract

The Gumbel-Softmax is a continuous distribution over the simplex that is often used as a relaxation of discrete distributions. Because it can be readily interpreted and easily reparameterized, it enjoys widespread use. We propose a modular and more flexible family of reparameterizable distributions where Gaussian noise is transformed into a one-hot approximation through an invertible function. This invertible function is composed of a modified softmax and can incorporate diverse transformations that serve different specific purposes. For example, the stick-breaking procedure allows us to extend the reparameterization trick to distributions with countably infinite support, thus enabling the use of our distribution along nonparametric models, or normalizing flows let us increase the flexibility of the distribution. Our construction enjoys theoretical advantages over the Gumbel-Softmax, such as closed form KL, and significantly outperforms it in a variety of experiments. Our code is available at https://github.com/cunningham-lab/igr.
BibTeX
@inproceedings{potapczynski2020invertible,
  author = {Andres Potapczynski and Gabriel Loaiza-Ganem and John P. Cunningham},
  title = {{Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=CRVan7Qd0u},
}
Figure