Stability Regularization for Discrete Representation Learning

Adeel Pervez, Efstratios GavvesICLR 2022 · OpenReview

Introduction

Training neural networks with discrete variables—such as binary switches, categories, or graph structures—is notoriously difficult because discrete operations break the flow of gradients required for backpropagation. This paper introduces Stability Regularization, a method rooted in Gaussian isoperimetric theory that pushes continuous representations toward discrete ones by maximizing their "noise stability." It offers a hassle-free alternative to existing methods, providing implicit temperature control and competitive performance across tasks ranging from physics simulations to image generation.

The Challenge of Discrete Variables

Deep learning thrives on differentiability, but many real-world computational tasks are inherently discrete. Whether learning a graph structure, gating redundant layers in a ResNet, or creating a compact bottleneck in an autoencoder, discrete variables are essential.

Standard workarounds have significant flaws:

  • Straight-Through Estimators: Simple but biased and often unstable in complex architectures.
  • Gumbel-Softmax: Requires careful manual tuning of a "temperature" parameter ($\tau$). If the temperature is too high, the model isn't discrete; if it's too low, gradients vanish.
  • Reinforcement Learning (REINFORCE): Unbiased but suffers from extremely high variance, making it slow and computationally expensive.

The authors seek a method that yields precise discrete representations without the "tuning tax" or the optimization instability of prior approaches.

Stability Regularization

The core innovation is Stability Regularization, based on a concept from Gaussian analysis: a function is "noise stable" if its output doesn't change much when a small amount of correlated noise is added to its input.

According to Borell’s Isoperimetric Theorem, the most noise-stable functions are "half-space indicators"—essentially binary switches. By adding a regularization term that rewards noise stability, the authors force the neural network to move away from fuzzy, continuous values and toward sharp, discrete outputs.

The paper proposes two main algorithms:

  1. Standard Stability Regularization: Samples two correlated Gaussian noise vectors, passes them through the layer, and maximizes the product of the outputs.
  2. Mean-Centered Stability Regularization: A more advanced version that prevents the inputs from drifting too far apart, which can otherwise stall optimization.

Crucially, this can be used as a standalone method or as a booster for Gumbel-Softmax to help it reach truly discrete states more reliably.

Key Insights and Results

Instead of forcing discreteness through a hard threshold or a specific distribution, this method treats discreteness as a byproduct of maximizing a geometric property (stability). This acts as an implicit temperature control, pushing the model toward discreteness naturally as part of the loss function without the extreme sensitivity to hyperparameters seen in other methods.

The authors validated their method on a diverse set of benchmarks, including MNIST, physics simulations (Neural Relational Inference), and CIFAR-10.

Log Probability Comparison
  • Structured Prediction: On MNIST, Stability Regularization outperformed Gumbel-Softmax, Rebar, and MuProp by a large margin in terms of validation log-probability.
  • Unsupervised Clustering: A simple Gaussian Mixture VAE with this regularization achieved 93.3% accuracy on MNIST, rivaling much more complex state-of-the-art models.
  • Physics Recovery: In Neural Relational Inference (NRI) tasks, adding stability regularization improved edge-recovery accuracy by 5-6% in charged particle systems.
  • Image Compression: Discrete autoencoders trained on CIFAR-10 achieved a 24x bit-size reduction. The resulting generative models achieved an FID score of 54.3, showing that the discrete latent space successfully captured global image structure.

Limitations and Future Work

While highly effective, the method does introduce a slight computational overhead, requiring the evaluation of the "stability layer" twice (once for each correlated noise sample). Furthermore, without the mean-centering variant or proper clipping, the regularization can push values so far into the tails of the sigmoid/softmax that gradients become too small to move, potentially slowing down late-stage optimization. Future work could explore how this approach scales to the massive discrete spaces found in Large Language Models (LLMs).

Summary updated

Abstract

We present a method for training neural network models with discrete stochastic variables. The core of the method is \emph{stability regularization}, which is a regularization procedure based on the idea of noise stability developed in Gaussian isoperimetric theory in the analysis of Gaussian functions. Stability regularization is method to make the output of continuous functions of Gaussian random variables close to discrete, that is binary or categorical, without the need for significant manual tuning. The method allows control over the extent to which a Gaussian function's output is close to discrete, thus allowing for continued flow of gradient. The method can be used standalone or in combination with existing continuous relaxation methods. We validate the method in a broad range of experiments using discrete variables including neural relational inference, generative modeling, clustering and conditional computing.
BibTeX
@inproceedings{pervez2022stability,
  author = {Adeel Pervez and Efstratios Gavves},
  title = {{Stability Regularization for Discrete Representation Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=6tmjoym9LR6},
}
Figure