Introduction
Training neural networks with discrete variables—such as binary switches, categories, or graph structures—is notoriously difficult because discrete operations break the flow of gradients required for backpropagation. This paper introduces Stability Regularization, a method rooted in Gaussian isoperimetric theory that pushes continuous representations toward discrete ones by maximizing their "noise stability." It offers a hassle-free alternative to existing methods, providing implicit temperature control and competitive performance across tasks ranging from physics simulations to image generation.
The Challenge of Discrete Variables
Deep learning thrives on differentiability, but many real-world computational tasks are inherently discrete. Whether learning a graph structure, gating redundant layers in a ResNet, or creating a compact bottleneck in an autoencoder, discrete variables are essential.
Standard workarounds have significant flaws:
- Straight-Through Estimators: Simple but biased and often unstable in complex architectures.
- Gumbel-Softmax: Requires careful manual tuning of a "temperature" parameter ($\tau$). If the temperature is too high, the model isn't discrete; if it's too low, gradients vanish.
- Reinforcement Learning (REINFORCE): Unbiased but suffers from extremely high variance, making it slow and computationally expensive.
The authors seek a method that yields precise discrete representations without the "tuning tax" or the optimization instability of prior approaches.
Stability Regularization
The core innovation is Stability Regularization, based on a concept from Gaussian analysis: a function is "noise stable" if its output doesn't change much when a small amount of correlated noise is added to its input.
According to Borell’s Isoperimetric Theorem, the most noise-stable functions are "half-space indicators"—essentially binary switches. By adding a regularization term that rewards noise stability, the authors force the neural network to move away from fuzzy, continuous values and toward sharp, discrete outputs.
The paper proposes two main algorithms:
- Standard Stability Regularization: Samples two correlated Gaussian noise vectors, passes them through the layer, and maximizes the product of the outputs.
- Mean-Centered Stability Regularization: A more advanced version that prevents the inputs from drifting too far apart, which can otherwise stall optimization.
Crucially, this can be used as a standalone method or as a booster for Gumbel-Softmax to help it reach truly discrete states more reliably.
Key Insights and Results
Instead of forcing discreteness through a hard threshold or a specific distribution, this method treats discreteness as a byproduct of maximizing a geometric property (stability). This acts as an implicit temperature control, pushing the model toward discreteness naturally as part of the loss function without the extreme sensitivity to hyperparameters seen in other methods.
The authors validated their method on a diverse set of benchmarks, including MNIST, physics simulations (Neural Relational Inference), and CIFAR-10.
- Structured Prediction: On MNIST, Stability Regularization outperformed Gumbel-Softmax, Rebar, and MuProp by a large margin in terms of validation log-probability.
- Unsupervised Clustering: A simple Gaussian Mixture VAE with this regularization achieved 93.3% accuracy on MNIST, rivaling much more complex state-of-the-art models.
- Physics Recovery: In Neural Relational Inference (NRI) tasks, adding stability regularization improved edge-recovery accuracy by 5-6% in charged particle systems.
- Image Compression: Discrete autoencoders trained on CIFAR-10 achieved a 24x bit-size reduction. The resulting generative models achieved an FID score of 54.3, showing that the discrete latent space successfully captured global image structure.
Limitations and Future Work
While highly effective, the method does introduce a slight computational overhead, requiring the evaluation of the "stability layer" twice (once for each correlated noise sample). Furthermore, without the mean-centering variant or proper clipping, the regularization can push values so far into the tails of the sigmoid/softmax that gradients become too small to move, potentially slowing down late-stage optimization. Future work could explore how this approach scales to the massive discrete spaces found in Large Language Models (LLMs).
Summary updated