Introduction
Active learning (AL) is a strategic approach to machine learning where a model identifies which unlabeled data points would be most beneficial to learn from next, aiming to achieve high accuracy with minimal labeled data. While highly effective in theory, applying AL to deep neural networks (NNs) presents significant challenges. Standard AL is computationally expensive, often requiring the model to be retrained after every new data point is added to update the "acquisition function." Furthermore, NNs are prone to overconfidence in regions far from their training data, leading to suboptimal sampling choices.
This paper introduces a novel framework that bridges the gap between the scalability of neural networks and the robust uncertainty quantification of Gaussian Processes (GPs). By approximating trained Bayesian Neural Networks (BNNs) with GPs—a concept known as the Neural Network Gaussian Process (NNGP)—the authors achieve more reliable uncertainty estimates and significantly faster active learning cycles.
Methodology: The NNGP Framework
The proposed approach leverages the theoretical connection that deep NNs with random weights converge to GPs in the infinite-width limit. The authors extend this to trained networks, showing they exhibit near-Gaussian behavior that can be exploited for efficient sampling.
The NNGP cycle consists of several key steps:
- BNN Approximation: The process begins with a Bayesian Neural Network, typically utilizing Monte Carlo Dropout.
- GP Construction: The stochastic output of the NN is treated as a random process and approximated as a GP. The covariance function (kernel) is estimated by taking Monte Carlo samples from the trained NN.
- Efficient Updates: Once the GP approximation is established, the algorithm uses the GP's posterior variance formula to update uncertainty estimates. Because this formula depends only on the locations of points rather than their labels, the algorithm can select a diverse batch of points without retraining the heavy NN at every step.
- Periodic Retraining: The computationally expensive NN is only retrained every $M$ steps. In the interim, the GP "simulates" how uncertainty would evolve as new points are added, preventing the selection of redundant, clustered data points.
Experimental Results
The authors evaluated the NNGP approach across diverse regression benchmarks, including the Airline Delays dataset, various UCI Machine Learning Repository datasets, and specialized scientific datasets like QM9 (quantum chemistry) and a physics-based Hydraulic Simulator.
Key performance highlights include:
- Quantum Chemistry (SchNet): The NNGP approach achieved a 25% increase in accuracy (RMSE) compared to standard dropout-based active learning. It demonstrated a 2x improvement in data efficiency, requiring only 7,500 additional molecules to reach a specific error threshold, compared to 15,000 for random sampling.
- Engineering (Hydraulic Simulator): NNGP outperformed baselines by a massive margin, yielding a maximum error ten times lower than competing methods.
- General Robustness: Across UCI datasets, the NNGP method consistently emerged as the top performer across various error tolerances, effectively identifying "unknown unknowns" that standard NNs often missed due to overconfidence.
Conclusion and Future Work
By combining the flexibility of neural networks with the elegant uncertainty updates of Gaussian Processes, this framework addresses the primary bottlenecks of active learning in deep learning. While the GP approximation introduces its own $O(N^3)$ complexity, the use of "anchor points" makes it a viable solution for accelerating the AL cycle.
Future research directions include extending this regression-focused framework to image classification tasks and developing more automated methods for anchor point selection to further enhance the scalability of the GP approximation.
Summary updated