Abstract
Active learning methods for neural networks are usually based on greedy criteria which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample. In this work we propose to approximate Bayesian neural networks (BNN) by Gaussian processes, which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network, while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data including large-scale problems of chemical and physical modeling, we show superiority of the proposed approach over the state-of-the-art methods.
Introduction
Active Learning (AL) is a critical strategy in domains where data acquisition is expensive, such as complex physical simulations or laboratory experiments. By allowing a model to select the most informative data points for labeling, AL aims to achieve high performance with minimal data. However, standard Neural Networks (NNs) face significant hurdles in AL: the high computational cost of frequent retraining, the tendency to select redundant points in greedy batches, and the issue of overconfidence in out-of-sample regions.
Methodology: The NNGP Framework
To address these challenges, researchers have leveraged the theoretical connection where deep neural networks with random weights converge to Gaussian Processes (GPs) as layer width increases. The proposed NNGP (Neural Network Gaussian Process) framework treats a trained Bayesian Neural Network (BNN) as a stochastic process and approximates it using a GP proxy.
The core innovation lies in the analytical updates provided by the GP proxy. Unlike NNs, GPs offer a closed-form mathematical method to update uncertainty. When a new point is selected for labeling, the GP immediately calculates the resulting reduction in uncertainty across the design space without requiring a full retraining of the underlying NN. This allows for an "M-step NNGP" algorithm that efficiently selects diverse batches of $M$ points by updating the uncertainty map after each selection within the batch.
Key Insights
- Mathematical Diversity: Instead of relying on distance-based heuristics to prevent picking points too close together, NNGP ensures diversity through the GP's posterior variance formula. Once a point is "sampled," the variance in its vicinity drops automatically, guiding the next selection to a different, still-uncertain region.
- Trained Kernels: While previous work focused on untrained NNs, this framework uses "anchor points" from the training data to estimate a covariance function based on the NN's own internal representations. This ensures the GP is a high-fidelity reflection of the trained model's learned features.
- Efficiency Gains: The most computationally expensive parts—NN retraining and black-box oracle labeling—are only performed every $M$ steps. All intermediate logic for picking the "best" points happens in the computationally cheaper GP space.
Experimental Results
The NNGP approach was evaluated across diverse regression tasks, including UCI benchmarks, airline delay datasets, and specialized simulations in chemistry and physics.
In chemistry simulations (SchNet on the QM9 dataset), NNGP achieved a target RMSE of 2 kcal/mol with 50% fewer samples than random sampling (7,500 vs. 15,000). In physics-based hydraulic simulations for oil recovery, NNGP's maximum error was ten times lower than random sampling and significantly outperformed standard Monte Carlo Dropout (MCDUE). Across UCI benchmarks, NNGP consistently emerged as the top-performing competitor compared to other state-of-the-art AL strategies.
Conclusion and Future Work
The NNGP framework provides a robust and efficient solution for active learning in regression tasks by bridging the gap between NNs and GPs. While the GP proxy introduces some scalability constraints due to $O(N^3)$ matrix inversion, it significantly reduces the overall computational burden of AL. Future research aims to extend this logic to non-stationary models and image classification tasks.
Summary updated