Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation

Mathias Löwe, Jes Frellsen, Per Lunnemann Hansen, Sebastian Risi

Abstract

We introduce a novel combination of Bayesian Models (BMs) and Neural Networks (NNs) for making predictions with a minimum expected risk. Our approach combines the best of both worlds: the data efficiency and interpretability of a BM with the speed of a NN. For a BM, making predictions with the lowest expected loss requires integrating over the posterior distribution. When exact inference is intractable, approximation methods like Monte Carlo (MC) simulation are typically applied. However, the computational cost of MC scales with the desired precision. Our approach removes the need for iterative MC simulation at prediction time by fitting a surrogate NN to synthetic data generated by the BM. Using Active Learning to minimize training data requirements, we achieve risk-minimized predictions significantly faster than standard methods with negligible loss in accuracy.

Introduction

In modern machine learning, practitioners often face a stark trade-off between data efficiency and computational speed. Bayesian Models (BMs) are highly valued for their ability to incorporate prior knowledge and provide robust uncertainty estimates, making them ideal for small-data regimes. However, their "risk-minimized" predictions—which require integrating over the entire posterior distribution—are computationally expensive, often relying on slow Monte Carlo simulations. This latency makes BMs difficult to deploy in real-time systems like autonomous vehicles or high-frequency trading platforms.

Conversely, Neural Networks (NNs) offer lightning-fast inference but are notoriously data-hungry and often struggle to provide reliable uncertainty estimates. This paper addresses the computational bottleneck of Bayesian prediction by asking: can we obtain the robust, risk-minimized predictions of a Bayesian model at the speed of a single neural network feed-forward pass?

Methodology

The proposed framework utilizes a three-step pipeline to create a fast neural approximation of a Bayesian model, as illustrated in Figure 1.

Conceptual framework showing the transition from observations to a Bayesian model, and finally to a surrogate neural network.
  1. Fit the Bayesian Model: A standard BM is first fitted to the available real-world dataset (which may be small) to obtain a posterior distribution over the model parameters ($\phi$).
  2. Generate Synthetic Data: The BM acts as a "teacher." We sample $M$ sets of parameters from the BM's posterior. For any input $x$, the BM can now produce a distribution of $M$ predictions.
  3. Train the Surrogate NN: A neural network $g$ is trained to map an input $x$ directly to a vector of $M$ values, approximating the BM's posterior predictive distribution.

Active Learning for Efficiency

To avoid the prohibitive cost of generating massive synthetic datasets, the authors employ Active Learning (AL). By using "MC Dropout" to measure the surrogate NN's predictive uncertainty, the algorithm iteratively identifies regions of the input space where the NN is most uncertain. It then labels only those specific points using the expensive Bayesian model, significantly reducing the number of BM evaluations required to achieve high approximation accuracy.

Experimental Setup

The methodology was evaluated using a synthetic Bayesian regression model designed to simulate varying levels of complexity: $$f(x) = \gamma + \sum_{j=1}^{J} \beta_j \psi(x_j \alpha_j)$$ where $\psi$ represents a non-linear activation function. By varying the number of features ($J$), the authors could observe how the BM and the surrogate NN scaled with increasing model complexity. The experiments utilized training, validation, and test sets of 50,000 examples each.

Results and Discussion

The empirical results demonstrate significant advantages in both speed and scalability.

Quantitative results showing prediction time, approximation error (MSE), and training set size requirements as a function of model complexity (J).

Speed and Accuracy

As shown in Figure 6, the prediction time for the Bayesian model grows linearly with complexity $J$. In contrast, the surrogate NN maintains a virtually constant inference time regardless of the underlying BM's complexity. For the most complex models tested, the NN achieved a speedup of over 14x compared to standard Bayesian inference. Crucially, this speedup comes with negligible loss in accuracy; the Mean Squared Error (MSE) on the test set remained between $4.3 \times 10^{-5}$ and $2.1 \times 10^{-4}$.

The Complexity Paradox

A surprising finding was the negative correlation between model complexity and approximation error. As $J$ increased, the NN actually found it easier to approximate the BM. This "Complexity Paradox" is attributed to the Central Limit Theorem: as more independent components are summed, the resulting distribution becomes smoother and more "normal," making it a simpler target for the NN to learn.

Data Efficiency

The integration of Active Learning proved highly effective. Doubling the complexity of the underlying Bayesian model required only a marginal increase in the training set size for the NN, rather than an exponential increase in data.

Conclusion

This research demonstrates that surrogate neural networks can effectively "distill" the predictive power of complex Bayesian models into a high-speed inference engine. By leveraging Active Learning, this approximation can be achieved with high data efficiency. Future work will explore the application of this method to dynamic models that update over time and more complex, "jagged" real-world distributions where the Central Limit Theorem may not provide the same smoothing benefits.

Summary updated

Abstract

We introduce a novel combination of Bayesian Models (BMs) and Neural Networks (NNs) for making predictions with a minimum expected risk. Our approach combines the best of both worlds, the data efficiency and interpretability of a BM with the speed of a NN. For a BM, making predictions with the lowest expected loss requires integrating over the posterior distribution. When exact inference of the posterior predictive distribution is intractable, approximation methods are typically applied, e.g. Monte Carlo (MC) simulation. For MC, the variance of the estimator decreases with the number of samples - but at the expense of increased computational cost. Our approach removes the need for iterative MC simulation on the CPU at prediction time. In brief, it works by fitting a NN to synthetic data generated using the BM. In a single feed-forward pass, the NN gives a set of point-wise approximations to the BM's posterior predictive distribution for a given observation. We achieve risk minimized predictions significantly faster than standard methods with a negligible loss on the test dataset. We combine this approach with Active Learning to minimize the amount of data required for fitting the NN. This is done by iteratively labeling more data in regions with high predictive uncertainty of the NN.
BibTeX
@inproceedings{lowe2021rapid,
  author = {Mathias Löwe and Jes Frellsen and Per Lunnemann Hansen and Sebastian Risi},
  title = {{Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=wikuLzUBYRK},
}
Figure