Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes

Jian Xu, Shian Du, Junmei Yang, Qianli Ma, Delu Zeng

TL;DR

Neural Operator Variational Inference (NOVI) is a new framework for Deep Gaussian Processes (DGPs) that replaces rigid Gaussian posterior assumptions with a flexible neural generator. By minimizing the Regularized Stein Discrepancy (RSD), NOVI achieves state-of-the-art accuracy on benchmarks like CIFAR-10 (93.56%) while converging significantly faster than traditional variational or MCMC-based methods.

What Problem Does This Paper Address?

Deep Gaussian Processes (DGPs) are powerful hierarchical models known for robust uncertainty quantification. However, their "exact" inference is mathematically intractable. Researchers typically rely on two main workarounds, both of which have flaws:

  1. Mean-Field Variational Inference (VI): This assumes the posterior distribution of the model's "inducing points" is a simple Gaussian. While fast, this assumption is often too restrictive, failing to capture the complex, non-Gaussian nature of the true posterior.
  2. Stochastic Gradient Hamiltonian Monte Carlo (SGHMC): This uses sampling to avoid Gaussian assumptions but is computationally expensive and difficult to monitor for convergence.

The authors aim to bridge this gap by creating an inference method that is as flexible as sampling but as scalable as variational inference.

Methods

The core of the paper is Neural Operator Variational Inference (NOVI). Instead of defining a specific functional form for the posterior (like a Gaussian), the authors use a neural network as a generator. This generator transforms simple noise into samples that represent the posterior distribution of the inducing variables.

To train this generator, they use Regularized Stein Discrepancy (RSD). Unlike the standard Kullback-Leibler (KL) divergence used in VI, Stein discrepancy doesn't require the probability density to be tractable—it only requires the "score function" (the gradient of the log-posterior).

Key technical innovations include:

  • Minimax Optimization: The training is set up as a game between a discriminator (trying to find the maximum discrepancy) and the generator (trying to minimize it).
  • Hutchinson Estimator: To keep the math efficient, they use this stochastic estimator to compute the trace of the Jacobian, reducing computational complexity from $O(D^2)$ to $O(D)$.
  • Fisher Divergence Connection: The authors provide theoretical proof that their training objective is equivalent to minimizing the Fisher divergence, which allows for robust error control and stable training.

Datasets

The authors evaluated NOVI across a wide range of tasks:

  • UCI Regression Benchmarks: Eight datasets including Boston, Energy, and Protein (ranging from hundreds to ~45,000 points).
  • Large-Scale Regression: The YearMSD (~500,000 points) and Airline (~2 million points) datasets.
  • Image Classification: MNIST, Fashion-MNIST, and CIFAR-10.
  • UCI Classification: German, Ringnorm, and Twonorm.

Key Insights and Ideas

  • Breaking the Gaussian Ceiling: By using a neural generator, the model can capture multi-modal or highly skewed posterior distributions that a standard Gaussian approximation would simply smooth over.
  • Efficiency through Operators: The use of the Langevin-Stein operator allows the model to utilize high-order geometric information (like curvature) of the distribution, leading to faster convergence.
  • Theoretical Guarantees: The paper proves that the bias in prediction can be bounded by the square root of the Fisher divergence, providing a mathematical safety net for the neural approximation.
  • Scalability to Millions: The combination of mini-batching, inducing points, and the Hutchinson estimator allows this flexible Bayesian approach to work on datasets with millions of entries, a traditional weak point for GPs.

Key Results

NOVI consistently outperformed state-of-the-art (SOTA) methods like Doubly Stochastic VI (DSVI) and SGHMC.

  • Accuracy: On CIFAR-10, NOVI achieved 93.56% accuracy, surpassing other Gaussian Process-based methods.
  • Convergence Speed: As shown in the ablation studies, NOVI often reaches a stable, high-performing state in just 500 iterations, whereas traditional methods like DSVI often require 20,000 iterations to reach comparable results.

![Fig.

  1. Regression mean test RMSE results. NOVI (blue) consistently shows lower error across various UCI datasets compared to SGHMC (orange), IPVI (pink), and DSVI (cyan).](/figures/art_53cbd260f2c34f5aa9f1d2226992b925)
  2. Overfitting Resistance: The adversarial nature of the Stein discrepancy training acts as a form of regularization. In regression tasks, NOVI achieved lower test RMSE (Root Mean Square Error) than direct log-likelihood maximization, indicating better generalization.

![Fig.

  1. Convergence comparison. NOVI (orange/red) converges much faster and to a lower error than the standard Monte Carlo log-likelihood method (blue/cyan).](/figures/art_2ca59863babd461d93e8f94065685278)

Limitations and Future Work

  • Layer Sensitivity: The authors noted that while NOVI is generally superior, performance can sometimes fluctuate as the number of DGP layers increases. Finding the "sweet spot" for depth remains a challenge.
  • Manual Architecture Selection: The generator and discriminator networks were designed manually. The authors suggest that Neural Architecture Search (NAS) could be used in the future to automatically find the best network structures for specific datasets.
  • Hyperparameter Tuning: The method introduces new hyperparameters, such as the regularization strength $\lambda$, which require careful tuning to balance training stability and posterior flexibility.

Summary updated

Abstract

Deep Gaussian Process (DGP) models offer a powerful nonparametric approach for Bayesian inference, but exact inference is typically intractable, motivating the use of various approximations. However, existing approaches, such as mean-field Gaussian assumptions, limit the expressiveness and efficacy of DGP models, while stochastic approximation can be computationally expensive. To tackle these challenges, we introduce Neural Operator Variational Inference (NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a sampler and minimizes the Regularized Stein Discrepancy in L2 space between the generated distribution and true posterior. We solve the minimax problem using Monte Carlo estimation and subsampling stochastic optimization techniques. We demonstrate that the bias introduced by our method can be controlled by multiplying the Fisher divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm. Our experiments on datasets ranging from hundreds to tens of thousands demonstrate the effectiveness and the faster convergence rate of the proposed method. We achieve a classification accuracy of 93.56 on the CIFAR10 dataset, outperforming SOTA Gaussian process methods. Furthermore, our method guarantees theoretically controlled prediction error for DGP models and demonstrates remarkable performance on various datasets. We are optimistic that NOVI has the potential to enhance the performance of deep Bayesian nonparametric models and could have significant implications for various practical applications
BibTeX
@article{xu2025neural,
  author = {Jian Xu and Shian Du and Junmei Yang and Qianli Ma and Delu Zeng},
  title = {{Neural Operator Variational Inference based on Regularized Stein Discrepancy for Deep Gaussian Processes}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2024.3406635},
}
Figure