Predictive Uncertainty Quantification for Safety-Critical Systems

167 concepts161 papers

Deep neural networks are incredibly capable, but they suffer from a dangerous flaw: they are notoriously overconfident. A standard deep learning model will happily predict a clear path for an autonomous vehicle or issue a medical diagnosis with 99% softmax probability, even when the input is completely unfamiliar or corrupted by noise.

In safety-critical domains, a point estimate without an accurate measure of confidence is a liability. To safely deploy these models, systems must be able to quantify their own predictive uncertainty. This shifts the engineering paradigm from verifying internal deterministic logic to statistically bounding the reliability of model outputs (Cazorla et al.). If a model can reliably signal when it is uncertain, a downstream controller can trigger fail-safe protocols, slow down, or hand control back to a human operator (Nair et al.).

The Anatomy of Uncertainty

To effectively manage uncertainty, we must first understand its origins. As formalized by frameworks like the DEA methodology (Standardizing the Probabilistic Sources of Uncertainty for the sake of Safety Deep Learning), predictive uncertainty is broadly split into two categories:

  1. Aleatoric Uncertainty: This represents the inherent randomness or noise in the environment (e.g., sensor noise, poor lighting, or motion blur). Even with infinite training data, aleatoric uncertainty cannot be eliminated. Kofnov demonstrates that tracking how this input noise propagates through a network's layers is essential for establishing exact error guarantees on the final output distribution.
  2. Epistemic Uncertainty: This represents a lack of knowledge about the underlying system, typically caused by a lack of training data. It answers the question: "Has the model ever seen something like this before?" When a model encounters out-of-distribution (OOD) data, epistemic uncertainty spikes.

Distinguishing between these two is vital for safety-critical decision-making. For instance, if an autonomous vehicle encounters a pedestrian obscured by heavy fog (high aleatoric uncertainty), it should cautiously proceed while gathering more sensor data. If it encounters an entirely novel object it cannot classify (high epistemic uncertainty), it must recognize its own ignorance and safely abstain from making a blind guess.

Sampling Methods and the Quest for Efficiency

The classical approach to capturing uncertainty relies on Bayesian formulations. Instead of learning a single fixed weight for each connection in a neural network, Bayesian Neural Networks (BNNs) learn a probability distribution over the weights. By sampling different weight configurations, the model generates an ensemble of predictions. The variance across these predictions serves as a direct proxy for uncertainty. For example, Morelli et al. utilize BNNs to provide real-time confidence bounds for in-flight ice detection on rotorcraft, ensuring pilots know exactly how much to trust the automated system.

Despite their theoretical elegance, sampling methods like Monte Carlo (MC) Dropout or Deep Ensembles are computationally expensive. Running a model 10 to 50 times for a single frame is a non-starter for real-time applications like robotics or autonomous driving.

To bridge this gap, recent research focuses on making sampling-based uncertainty computationally viable:

  • Adaptive Sampling: Bethell et al. introduce a framework that monitors the convergence of predictions during the sampling process. By dynamically halting the Monte Carlo simulations once the prediction stabilizes, they cut inference time in half without sacrificing interval accuracy.
  • Feature-Map Ensembling: Rather than duplicating the entire network pipeline, methods like LiDAR-MIMO (Pitropov et al.) apply multi-input multi-output architectures directly to internal feature maps, achieving ensemble-grade uncertainty for 3D object detection at a fraction of the computational cost.
  • Asymmetric Duos: Zhou et al. propose pairing a massive, high-performance "base" model with a tiny, computationally cheap "sidekick" model. By learning to weight the disagreement between the two, this duo approximates full ensemble uncertainty with only a minor increase in compute overhead.

Single-Shot and Post-Hoc Calibration

An alternative strategy avoids sampling entirely by forcing the network to output uncertainty parameters in a single forward pass, or by extracting uncertainty post hoc from a pre-trained model.

Evidential Deep Learning

Instead of predicting class probabilities directly via a softmax layer, Evidential Deep Learning (EDL) treats the network's outputs as parameters of a higher-order probability distribution. Sensoy et al. place a Dirichlet distribution over the class probabilities. The network learns to output "evidence" metrics for each class. If the network lacks evidence for any option, the Dirichlet distribution flattens, explicitly capturing epistemic uncertainty in a single shot.

Auxiliary and Gradient-Based Metacognition

Rather than altering the core architecture, researchers also design auxiliary systems to monitor the primary model. Corbière introduces ConfidNet, an auxiliary network trained specifically to predict the True Class Probability (TCP) of the main model's predictions, bypassing the flawed maximum softmax probability.

We can also look at the internal state of the network itself to gauge confidence. When a model is presented with an unfamiliar image, the internal gradients required to process or reconcile that image behave differently than they do for familiar data. Hornauer et al. leverage this intuition for monocular depth estimation, calculating pixel-wise uncertainty by evaluating gradients from an auxiliary consistency loss without needing ground-truth data during inference. This builds on the foundational concept that network-wide gradient magnitudes reflect "re-learning stress," making them highly sensitive indicators of misclassifications and OOD samples (Oberdiek et al.).

Standard Softmax:   [Input] ---> [Deep Network] -----------> [Overconfident Class Probabilities]
Evidential (EDL):   [Input] ---> [Deep Network] -----------> [Dirichlet Parameters (Evidence)] ---> [Explicit Uncertainty]
Auxiliary Network:  [Input] ---> [Deep Network] ---> [TCP] -> [ConfidNet (Predicts Failure Risk)]

Conformal Prediction: Rigorous Statistical Guarantees

While heuristics, calibration toolkits (Hsiung et al.), and Bayesian approximations improve a model's relative confidence ranking, they do not provide formal mathematical guarantees. If a safety standard requires that an object detection bounding box must contain the actual obstacle 99% of the time, heuristic confidence scores are insufficient.

This requirement has led to the widespread adoption of Conformal Prediction (CP). Conformal prediction is a user-defined, distribution-free framework that takes a heuristic uncertainty score and maps it to a rigorous prediction region (or set) that satisfies a strict statistical coverage guarantee:

$$\mathbb{P}(Y \in \hat{C}(X)) \ge 1 - \alpha$$

where $\hat{C}(X)$ is the prediction interval and $\alpha$ is the user-specified error tolerance (e.g., $\alpha = 0.05$ for 95% coverage).

Scaling and Adapting Conformal Inference

Traditional "Full" Conformal Prediction is computationally impossible for deep learning because it requires retraining the model for every single test point. To make this practical, Abad Martinez et al. utilize influence functions to simulate model retraining instantly, scaling full CP guarantees to deep neural networks.

Furthermore, researchers are actively adapting conformal inference to handle the complex, multi-dimensional outputs characteristic of safety-critical systems:

  • Multi-Object Detection: Standard conformal prediction struggles when an object is misclassified, as the spatial uncertainty bounds break down. Timans et al. resolve this by introducing a two-step conformal framework. It first generates a conformal set of valid class labels, and then propagates that semantic uncertainty directly into the bounding box regression, ensuring the physical safety margins remain mathematically valid.
  • Dynamic Time Series: In sequential environments, such as trajectory forecasting or autonomous navigation, errors compound over time. Cleaveland et al. replace loose, overly conservative uncertainty bounds with parameterized scoring functions optimized via Linear Complementarity Programming, producing tight, highly usable "uncertainty bubbles" around moving agents. This data-driven bounding allows engineers to perform rigorous reachability analysis on complex stochastic systems without knowing their underlying mathematical equations (Hashemi et al.).

By combining the raw predictive power of deep neural networks with the rigorous mathematical guardrails of conformal inference and efficient single-shot calibration, modern safety-critical systems can transition from blind execution to uncertainty-aware, risk-mitigating autonomy.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_uncertainty_quantification_in_safety_cri_f81c3bf4,
  title = {uncertainty quantification in safety-critical systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-in-safety-critical-systems/rdr_f81c3bf4732d411494409b8106782e1f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_uncertainty_quantification_in_safety_cri_f81c3bf4_v1,
  title = {uncertainty quantification in safety-critical systems},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/uncertainty-quantification-in-safety-critical-systems/rdr_f81c3bf4732d411494409b8106782e1f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake