Neural Stein Critics With Staged L 2-Regularization

Matthew Repasky, Xiuyuan Cheng, Yao Xie

TL;DR

This paper introduces a novel "staged regularization" technique for training neural Stein critics, which are powerful tools for measuring discrepancies between probability distributions. By dynamically adjusting the regularization strength during training, they achieve better performance and efficiency, particularly in high-dimensional settings, and provide theoretical guarantees for their method.

What Problem Does This Paper Address?

Understanding and quantifying the differences between probability distributions is a fundamental challenge in machine learning and statistics. This is crucial for tasks like evaluating generative models (e.g., GANs), detecting out-of-distribution samples, and performing goodness-of-fit tests. Traditional methods often struggle with high-dimensional data or require knowledge of normalizing constants, which are difficult to compute. The Stein discrepancy offers a promising alternative as it only requires the score function (gradient of the log-density) of the model distribution. However, effectively training neural networks to act as "Stein critics" for this discrepancy, especially in high dimensions, remains an open problem. This paper specifically addresses how to optimize the training of these neural Stein critics by carefully managing the regularization process.

Methods

The core contribution is a staged L²-regularization strategy for training neural Stein critics. Instead of using a fixed regularization parameter ($\lambda$) throughout training, the authors propose to start with a large $\lambda$ and gradually decrease it. This strategy is motivated by theoretical connections to the "lazy training" phenomenon in neural networks, where large regularization at the beginning leads to a kernel regression-like behavior.

Key aspects of their method include:

  • Neural Stein Critics: Parameterized neural networks that learn to approximate the optimal critic function for the Stein discrepancy.
  • L²-Regularization: A standard technique to prevent overfitting and improve generalization, applied to the critic's objective function.
  • Staged Regularization: The novel approach where $\lambda$ is annealed over training epochs. This is implemented using a log-linear decay schedule based on initial ($\lambda_{init}$), terminal ($\lambda_{term}$), and decay factor ($\beta$) parameters.
  • Theoretical Analysis: The paper provides theoretical guarantees, connecting the staged regularization to the Neural Tangent Kernel (NTK) theory and proving convergence rates under certain assumptions. They show that large $\lambda$ early in training leads to a kernel learning dynamic, which is beneficial for initial approximation.

Datasets

The paper uses simulated datasets for evaluation:

  • 1D Gaussian Mixtures: Used for initial visualization and understanding of the staged regularization.
  • High-dimensional Gaussian Mixtures (2D, 10D, 25D): Used to demonstrate the effectiveness of the staged approach in increasing dimensions.
  • MNIST Handwritten Digits Dataset: Used for a real-world application, comparing the trained critics to a Gaussian-Bernoulli RBM model.

Key Insights and Ideas

  • Dynamic Regularization is Key: The "one-size-fits-all" approach of fixed regularization is suboptimal. Dynamically adjusting $\lambda$ during training unlocks significant performance gains, especially in high dimensions.
  • Early Large Regularization for Fast Approximation: Starting with a high $\lambda$ leverages the NTK "lazy training" regime, allowing the critic to quickly approximate the target function. This is crucial for efficient learning.
  • Gradual Annealing for Fine-Tuning: Decreasing $\lambda$ later in training allows the critic to refine its approximation and avoid overfitting, leading to a more accurate representation of the distribution discrepancy.
  • Theoretical Justification for Empirical Success: The connection to NTK theory provides a solid theoretical foundation for why staged regularization works, bridging the gap between empirical observations and mathematical understanding.
  • Scaleless Optimal Critic: The concept of a "scaleless optimal critic" ($f^*$) is introduced, which helps in analyzing the training dynamics and understanding the role of $\lambda$ as a scaling parameter.

Key Results

  • Improved GoF Testing Power: In high-dimensional Gaussian mixture settings (up to 25D), staged regularization consistently outperformed fixed $\lambda$ strategies in terms of GoF hypothesis testing power. For instance, in 25D, staged critics achieved dramatically higher power than fixed- $\lambda$ training.
  • Lower MSEq: The staged approach generally resulted in lower Mean Squared Error (MSEq) values, indicating a better approximation of the optimal critic.
  • Efficiency Gains: While training time for neural critics is still a factor, the staged approach leads to more efficient learning, achieving better results in fewer epochs or with less computational effort compared to some fixed $\lambda$ strategies.
  • Comparison to Kernel Stein Discrepancy (KSD): The neural Stein critic GoF test demonstrated superior performance (higher power) and better scalability in high dimensions compared to KSD, especially for larger sample sizes.
  • Real-world Application (MNIST): On the MNIST dataset, staged regularization showed comparable performance to the best fixed- $\lambda$ strategies, with a more rapid increase in validation metrics in the early training period.

Limitations and Future Work

  • Theoretical Analysis Complexity: The theoretical analysis primarily focuses on the "lazy training" regime (large $\lambda$) and finite-width networks. Extending this to analyze the full training dynamic, especially with decreasing $\lambda$, remains a challenge.
  • Generalization of Regularization Types: The paper focuses on L²-regularization. Exploring other regularization techniques or combinations could be beneficial.
  • Adaptive Staging: While the paper proposes a heuristic staged approach, adaptive staging methods (e.g., based on validation error) were investigated but found to be computationally more expensive without significant performance gains over the heuristic.
  • Broader Applications: The authors suggest extending the framework to other generative modeling approaches (e.g., normalizing flows) and exploring more complex GoF testing scenarios (e.g., composite null hypotheses).

Summary updated

Abstract

Learning to differentiate model distributions from observed data is a fundamental problem in statistics and machine learning, and high-dimensional data remains a challenging setting for such problems. Metrics that quantify the disparity in probability distributions, such as the Stein discrepancy, play an important role in high-dimensional statistical testing. In this paper, we investigate the role of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L^{2}$ </tex-math></inline-formula> regularization in training a neural network Stein critic so as to distinguish between data sampled from an unknown probability distribution and a nominal model distribution. Making a connection to the Neural Tangent Kernel (NTK) theory, we develop a novel staging procedure for the weight of regularization over training time, which leverages the advantages of highly-regularized training at early times. Theoretically, we prove the approximation of the training dynamic by the kernel optimization, namely the “lazy training”, when the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L^{2}$ </tex-math></inline-formula> regularization weight is large, and training on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> samples converge at a rate of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${O}(n^{-1/2})$ </tex-math></inline-formula> up to a log factor. The result guarantees learning the optimal critic assuming sufficient alignment with the leading eigen-modes of the zero-time NTK. The benefit of the staged <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$L^{2}$ </tex-math></inline-formula> regularization is demonstrated on simulated high dimensional data and an application to evaluating generative models of image data.
BibTeX
@article{repasky2023neural,
  author = {Matthew Repasky and Xiuyuan Cheng and Yao Xie},
  title = {{Neural Stein Critics With Staged L 2-Regularization}},
  journal = {IEEE Trans. Information Theory},
  year = {2023},
  doi = {10.1109/tit.2023.3299258},
}
Figure