Neural Stein critics with staged L2-regularization

Matthew Repasky, Xiuyuan Cheng, Yao Xie

Abstract

Learning to differentiate model distributions from observed data is a fundamental problem in statistics and machine learning, and high-dimensional data remains a challenging setting for such problems. Metrics that quantify the disparity in probability distributions, such as the Stein discrepancy, play an important role in high-dimensional statistical testing. In this paper, we investigate the role of $L^2$ regularization in training a neural network Stein critic so as to distinguish between data sampled from an unknown probability distribution and a nominal model distribution. Making a connection to the Neural Tangent Kernel (NTK) theory, we develop a novel staging procedure for the weight of regularization over training time, which leverages the advantages of highly-regularized training at early times. Theoretically, we prove the approximation of the training dynamic by the kernel optimization, namely the "lazy training", when the $L^2$ regularization weight is large, and training on $n$ samples converge at a rate of ${O}(n^{-1/2})$ up to a log factor. The result guarantees learning the optimal critic assuming sufficient alignment with the leading eigen-modes of the zero-time NTK. The benefit of the staged $L^2$ regularization is demonstrated on simulated high dimensional data and an application to evaluating generative models of image data.

Introduction

Goodness-of-Fit (GoF) testing is a cornerstone of statistical inference, tasked with determining if observed data originates from a specific nominal model distribution. This task is increasingly critical in modern machine learning for evaluating generative models, such as GANs or Energy-Based Models, and for out-of-distribution detection.

The Stein discrepancy offers a powerful framework for GoF testing because it bypasses the need to compute the model's normalizing constant—a significant hurdle in high-dimensional spaces. While Kernel Stein Discrepancy (KSD) is a standard approach, it often suffers from the curse of dimensionality and quadratic computational scaling. Neural Stein Critics provide a flexible alternative but are notoriously difficult to optimize. This work introduces a staged $L^2$-regularization strategy to stabilize and enhance the training of these neural critics.

Staged $L^2$-Regularization and Lazy Training

The proposed method utilizes a neural network $f(x, \theta)$ as a Stein critic, trained with a time-varying $L^2$ regularization weight $\lambda$. The core innovation is a "staged" schedule: starting with a large $\lambda$ (strong penalty) and exponentially decaying it over time to a smaller terminal value.

This approach is grounded in Neural Tangent Kernel (NTK) theory. Under high regularization, the neural network enters a "lazy training" regime where its dynamics are well-approximated by a linear kernel method. This regime allows for rapid initial convergence ($O(n^{-1/2})$ rate) toward a "rough draft" of the optimal critic. As $\lambda$ decreases, the network transitions out of the lazy regime, utilizing its full non-linear expressivity to capture subtle distributional differences.

Training dynamics of the neural Stein critic under staged $L^2$-regularization. Top row (A-C) shows the convergence of the critic $f$ to the regularized optimal $f^*_\lambda$. Bottom row (D-F) shows the scaled critic $\lambda f$ approaching the unregularized optimal $f^*$.

As illustrated in the figure above, at early epochs with high $\lambda$, the critic quickly aligns with the regularized target. As training progresses and $\lambda$ decays, the approximation of the unregularized optimal critic $f^*$ improves significantly, demonstrating the effectiveness of the staging procedure.

Experimental Evaluation

The effectiveness of staged regularization was tested across several benchmarks:

  • High-Dimensional Scaling: In tests ranging from 1D to 50D, the staged neural critic significantly outperformed standard KSD. Notably, while KSD's cost grows quadratically ($O(n^2)$), the neural critic's evaluation cost is linear ($O(n)$), becoming more time-efficient for sample sizes exceeding 1,500.
  • MNIST Generative Modeling: The method distinguished authentic MNIST digits from those generated by a Restricted Boltzmann Machine (RBM). The "witness function" allowed for visualizing exactly where the model failed to match the data distribution.
  • Sensitivity to Contamination: In a challenging scenario with 3% data contamination, the staged critic demonstrated high sensitivity to local departures that traditional metrics often smooth over.

Conclusion

Staged $L^2$-regularization bridges the gap between the stability of kernel methods and the expressivity of neural networks. By leveraging "lazy" dynamics early in training, the neural Stein critic achieves faster convergence and superior sensitivity in high-dimensional GoF tests. While the current exponential decay schedule is heuristic, the theoretical grounding in NTK theory provides a robust path forward for developing more advanced training strategies for neural-based statistical tests.

Summary updated

Abstract

Learning to differentiate model distributions from observed data is a fundamental problem in statistics and machine learning, and high-dimensional data remains a challenging setting for such problems. Metrics that quantify the disparity in probability distributions, such as the Stein discrepancy, play an important role in high-dimensional statistical testing. In this paper, we investigate the role of $L^2$ regularization in training a neural network Stein critic so as to distinguish between data sampled from an unknown probability distribution and a nominal model distribution. Making a connection to the Neural Tangent Kernel (NTK) theory, we develop a novel staging procedure for the weight of regularization over training time, which leverages the advantages of highly-regularized training at early times. Theoretically, we prove the approximation of the training dynamic by the kernel optimization, namely the ``lazy training'', when the $L^2$ regularization weight is large, and training on $n$ samples converge at a rate of ${O}(n^{-1/2})$ up to a log factor. The result guarantees learning the optimal critic assuming sufficient alignment with the leading eigen-modes of the zero-time NTK. The benefit of the staged $L^2$ regularization is demonstrated on simulated high dimensional data and an application to evaluating generative models of image data.
BibTeX
@article{repasky2023neural,
  author = {Matthew Repasky and Xiuyuan Cheng and Yao Xie},
  title = {{Neural Stein critics with staged L2-regularization}},
  journal = {IEEE Trans. Information Theory},
  year = {2023},
  doi = {10.1109/tit.2023.3299258},
}
Figure