Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning

Evgenii Tsymbalov, Sergei Makarychev, Alexander Shapeev, Maxim Panov

Introduction

Active learning (AL) is a strategic approach to machine learning where a model identifies which unlabeled data points would be most beneficial to learn from next, aiming to achieve high accuracy with minimal labeled data. While highly effective in theory, applying AL to deep neural networks (NNs) presents significant challenges. Standard AL is computationally expensive, often requiring the model to be retrained after every new data point is added to update the "acquisition function." Furthermore, NNs are prone to overconfidence in regions far from their training data, leading to suboptimal sampling choices.

This paper introduces a novel framework that bridges the gap between the scalability of neural networks and the robust uncertainty quantification of Gaussian Processes (GPs). By approximating trained Bayesian Neural Networks (BNNs) with GPs—a concept known as the Neural Network Gaussian Process (NNGP)—the authors achieve more reliable uncertainty estimates and significantly faster active learning cycles.

Methodology: The NNGP Framework

The proposed approach leverages the theoretical connection that deep NNs with random weights converge to GPs in the infinite-width limit. The authors extend this to trained networks, showing they exhibit near-Gaussian behavior that can be exploited for efficient sampling.

The NNGP active learning cycle, illustrating the interaction between the trained neural network, Gaussian Process approximation, and the acquisition function.

The NNGP cycle consists of several key steps:

  1. BNN Approximation: The process begins with a Bayesian Neural Network, typically utilizing Monte Carlo Dropout.
  2. GP Construction: The stochastic output of the NN is treated as a random process and approximated as a GP. The covariance function (kernel) is estimated by taking Monte Carlo samples from the trained NN.
  3. Efficient Updates: Once the GP approximation is established, the algorithm uses the GP's posterior variance formula to update uncertainty estimates. Because this formula depends only on the locations of points rather than their labels, the algorithm can select a diverse batch of points without retraining the heavy NN at every step.
  4. Periodic Retraining: The computationally expensive NN is only retrained every $M$ steps. In the interim, the GP "simulates" how uncertainty would evolve as new points are added, preventing the selection of redundant, clustered data points.

Experimental Results

The authors evaluated the NNGP approach across diverse regression benchmarks, including the Airline Delays dataset, various UCI Machine Learning Repository datasets, and specialized scientific datasets like QM9 (quantum chemistry) and a physics-based Hydraulic Simulator.

Comparison of Test RMSE across different active learning algorithms as a function of data points seen, demonstrating the efficiency of NNGP-based methods.

Key performance highlights include:

  • Quantum Chemistry (SchNet): The NNGP approach achieved a 25% increase in accuracy (RMSE) compared to standard dropout-based active learning. It demonstrated a 2x improvement in data efficiency, requiring only 7,500 additional molecules to reach a specific error threshold, compared to 15,000 for random sampling.
  • Engineering (Hydraulic Simulator): NNGP outperformed baselines by a massive margin, yielding a maximum error ten times lower than competing methods.
  • General Robustness: Across UCI datasets, the NNGP method consistently emerged as the top performer across various error tolerances, effectively identifying "unknown unknowns" that standard NNs often missed due to overconfidence.

Conclusion and Future Work

By combining the flexibility of neural networks with the elegant uncertainty updates of Gaussian Processes, this framework addresses the primary bottlenecks of active learning in deep learning. While the GP approximation introduces its own $O(N^3)$ complexity, the use of "anchor points" makes it a viable solution for accelerating the AL cycle.

Future research directions include extending this regression-focused framework to image classification tasks and developing more automated methods for anchor point selection to further enhance the scalability of the GP approximation.

Summary updated

Abstract

Active learning methods for neural networks are usually based on greedy criteria which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample. In this work we propose to approximate Bayesian neural networks (BNN) by Gaussian processes, which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network, while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data including large-scale problems of chemical and physical modeling, we show superiority of the proposed approach over the state-of-the-art methods.
BibTeX
@inproceedings{tsymbalov2019deeper,
  author = {Evgenii Tsymbalov and Sergei Makarychev and Alexander Shapeev and Maxim Panov},
  title = {{Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=-wh5KQaQSa},
}
Figure