Abstract: Epistemic uncertainty quantification is a crucial part of drawing credible conclusions from predictive models, whether concerned about the prediction at a given point or any downstream evaluation that uses the model as input. When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies include the bootstrap, but it can be computationally infeasible when training the model even once is costly. In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of the predictive model to automatically assess downstream uncertainty. We show that the change in the evaluation due to regularization is consistent for the asymptotic variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference. This provides both a reliable quantification of uncertainty in terms of standard errors as well as permits the construction of calibrated confidence intervals. We discuss connections to other approaches to uncertainty quantification, both Bayesian and frequentist, and demonstrate our approach empirically.
Introduction: The Challenge of Uncertainty Quantification
Uncertainty quantification (UQ) is the critical difference between a model that provides a point estimate and one that provides a range of confidence. For simple, differentiable models, the Delta Method is the standard approach, propagating parameter uncertainty to downstream evaluations. However, for modern complex models like deep neural networks, the Delta Method requires inverting a massive Fisher Information Matrix, which is often computationally infeasible. Conversely, the Bootstrap—retraining the model hundreds of times—is prohibitively expensive for large-scale architectures that take hours or days to train once.
In "The Implicit Delta Method" (NeurIPS 2022), Nathan Kallus and James McInerney propose a middle ground: the Implicit Delta Method (IDM). IDM provides calibrated confidence intervals by training a model just one additional time, avoiding both the matrix-inversion nightmare of the Delta Method and the high retraining costs of the Bootstrap.
Methodology: Optimization as a Proxy for Calculus
The core innovation of IDM is to perform the Delta Method implicitly through regularization. Instead of explicitly calculating derivatives of the evaluation function $\psi(\theta)$ with respect to all parameters, IDM adds an infinitesimal "nudge" to the training objective.
If the goal is to estimate the uncertainty of an evaluation $\psi(\theta)$, the model is trained a second time with a modified objective: $$\text{Original Loss} + \lambda \psi(\theta)$$ where $\lambda$ is a small scalar. The authors prove that the infinitesimal change in the evaluation result as $\lambda$ moves away from zero is mathematically equivalent to the variance predicted by the classic Delta Method. In practice, the Finite-Difference IDM (FDIDM) uses the difference between the original evaluation $\psi(\hat{\theta}_n)$ and the regularized evaluation $\psi(\hat{\theta}_n(\lambda))$, scaled by $1/\lambda$, to estimate variance. This approach allows the optimizer itself to "calculate" the necessary derivatives by finding the direction of highest sensitivity.
Empirical Performance and Results
The authors validated IDM across synthetic datasets, UCI benchmarks, and the MNIST dataset using Variational Autoencoders (VAEs).
Computational Efficiency
On the MNIST dataset, IDM generated confidence intervals in approximately 303 seconds. In contrast, the Bootstrap required 7,164 seconds to achieve comparable results—a ~23x speedup. For a $K$-dimensional evaluation, IDM requires only $K+1$ training runs, making it significantly more scalable than traditional resampling methods for high-dimensional models.
Accuracy and Calibration
Synthetic tests demonstrated that IDM's confidence intervals closely track the "true" sampling uncertainty and the classic Delta Method. Notably, IDM outperformed Gaussian Processes in capturing uncertainty during extrapolation tasks. The method consistently produced calibrated intervals, meaning a 95% confidence interval actually contained the true value approximately 95% of the time in repeated trials.
Discussion and Limitations
While IDM offers a powerful shortcut for UQ, it relies on the model reaching a stable Maximum Likelihood Estimate (MLE). If the loss landscape is unstable or the model has not converged, the "nudge" may yield unreliable results. Additionally, like other asymptotic methods, IDM may be less accurate for very small datasets where the assumption of normality does not yet hold.
Despite these constraints, IDM bridges the gap between frequentist and Bayesian UQ. It can be viewed as a frequentist implementation of the Laplace approximation, providing the benefits of parameter covariance analysis without the need to explicitly compute or store the covariance matrix. This makes it a highly practical tool for uncertainty quantification in modern machine learning workflows.
Summary updated