Abstract
Deep Neural Networks (DNNs) often exhibit overconfidence, even when making incorrect predictions on out-of-distribution data. This paper presents a novel sampling-based Bayesian inference method that utilizes "gradient uncertainty" within a minibatch to trigger posterior sampling. By sampling only when the model is near a local optimum—identified by diverse, low-magnitude gradients—the approach efficiently captures predictive uncertainty while maintaining high accuracy and avoiding the computational overhead of continuous noise injection.
Introduction
Standard Deep Neural Networks are notoriously overconfident, a significant drawback for safety-critical applications like autonomous driving or medical diagnosis. Bayesian Neural Networks (BNNs) address this by treating weights as distributions. However, exact posterior inference is intractable, and existing approximations like Variational Inference or Markov Chain Monte Carlo (MCMC) can be computationally intensive or fail to capture parameter diversity. This work introduces a more efficient mechanism for deciding when to explore the parameter space via sampling.
Methodology: Gradient Uncertainty as a Sampling Trigger
The authors propose a metric called Gradient Uncertainty to decide when to sample parameters during training. This metric is defined as the sum of the inner products between gradient vectors of different examples within a single minibatch.
The Logic of Convergence
- Far from a solution: Gradients for different data points point in similar directions toward the local minimum, resulting in large inner products.
- Near a local optimum: The average gradient approaches zero, and individual gradients become "randomly scattered" or diverse. The sum of their inner products becomes very small, signaling that the model has reached a local mode.
The Algorithm
The training process proceeds as follows:
- Calculate gradients for a minibatch.
- Compute the Gradient Uncertainty (sum of inner products).
- If the value is below a predefined threshold:
- Save the current parameters (Sample).
- Update parameters by adding the mean gradient plus Gaussian noise to facilitate exploration of other modes.
- Otherwise, perform a standard gradient descent update.
Unlike Stochastic Gradient Langevin Dynamics (SGLD), which adds noise at every step, this method targets noise injection to regions where the model has converged, allowing for more efficient exploration of the loss landscape.
Experimental Evaluation
The method was evaluated on two benchmarks:
- MNIST: Handwritten digits (0-9) for in-distribution performance.
- notMNIST: Letters (A-J) used as out-of-distribution (OOD) data.
Key Results
The proposed method was compared against SGLD and Dropout, a common heuristic for uncertainty.
Quantitative Performance: On the notMNIST OOD task, the proposed method achieved a Predictive Entropy of 1.355, significantly outperforming SGLD (0.107) and Dropout (0.665). Higher entropy indicates better uncertainty quantification for unseen data. In-distribution accuracy on MNIST remained high at 98.05%, proving that uncertainty estimation does not compromise raw performance.
Qualitative Analysis: Visualizations of class probabilities for ambiguous digits demonstrate that the proposed method effectively captures uncertainty.
As seen in the figure, for confusing digits (e.g., a '7' that resembles a '2' or '3'), the proposed method spreads probability across multiple plausible classes. In contrast, baseline methods like Dropout often exhibit high confidence in a single class, even when that class is incorrect.
Discussion and Future Work
While promising, the method requires manual tuning of the gradient uncertainty threshold. Future research could explore automated threshold selection or adaptive learning rates based on the gradient uncertainty metric. Additionally, broader comparisons against state-of-the-art SG-MCMC and variational methods are necessary to fully characterize the approach's strengths in complex loss landscapes.
Summary updated