Bayesian Optimized Deep Ensemble for Uncertainty Quantification of Deep Neural Networks: a System Safety Case Study on Sodium Fast Reactor Thermal Stratification Modeling

Zaid Abulawi, Rui Hu, Prasanna Balaprakash, Yang Liu

TL;DR

The paper introduces BODE (Bayesian Optimized Deep Ensemble), a method that significantly improves both the predictive accuracy and uncertainty quantification (UQ) of deep neural networks. By using Bayesian Optimization to tune the hyperparameters of each individual ensemble member, BODE reduces total uncertainty by up to four times compared to standard ensembles and accurately distinguishes between data noise (aleatoric) and model ignorance (epistemic).

What Problem Does This Paper Address?

In safety-critical fields like nuclear engineering, knowing how much to trust a model is just as important as the prediction itself. Deep Neural Networks (DNNs) are increasingly used to replace slow, expensive simulations (like Computational Fluid Dynamics), but they often struggle with reliable Uncertainty Quantification.

The current "gold standard" for UQ in deep learning is the Deep Ensemble (DE)—training the same model multiple times with different random initializations. However, the authors point out a major flaw: standard ensembles are often "lazy." Because they use the same architecture and hyperparameters for every member, they lack the diversity needed to provide high-quality UQ. This often leads to the model "hallucinating" high aleatoric uncertainty (inherent data noise) even when the training data is perfectly clean.

Methods

The authors propose BODE, which integrates Bayesian Optimization (BO) into the ensemble construction process.

  1. Hyperparameter Diversity: Instead of retraining the same model, BODE uses BO to find the optimal set of hyperparameters (learning rate, batch size, dropout, etc.) for each member of the ensemble. This ensures the ensemble is composed of diverse, high-performing models.
  2. Optimization Strategy: They used the Ax-platform and BoTorch, starting with a Sobol sequence (a quasi-random sampling method) to explore the search space, followed by BO to exploit the best configurations.
  3. Loss Function: The models are trained using Negative Log-Likelihood (NLL) assuming a Gaussian distribution. This allows the network to output both a mean prediction and a variance (representing aleatoric uncertainty).
  4. UQ Decomposition: The total uncertainty is mathematically split into:
  5. Aleatoric: Uncertainty due to noise in the data.
  6. Epistemic: Uncertainty due to the model's lack of knowledge (measured by the variance between ensemble members).

Datasets

The study focuses on a Sodium Fast Reactor (SFR) case study, specifically modeling thermal stratification during a loss-of-flow event.

  • Source: High-fidelity CFD data generated via STAR-CCM+.
  • Input Features: Temperature, pressure, and velocity components.
  • Target: Eddy viscosity (a key parameter for turbulence modeling).
  • Preprocessing: The unstructured CFD mesh was converted to a structured $32 \times 64 \times 128$ grid using a k-Nearest Neighbor (kNN) algorithm.
  • Synthetic Noise: To test UQ robustness, the authors manually injected 5% and 10% Gaussian noise into the otherwise "clean" CFD results.

Key Insights and Ideas

  • Diversity Improves Confidence: By forcing ensemble members to have different architectures and training dynamics, BODE captures a much more realistic "model disagreement," which leads to lower and more accurate epistemic uncertainty.
  • Correcting Aleatoric Bias: A major "aha moment" was seeing the baseline ensemble overestimate aleatoric uncertainty in noise-free data. BODE correctly pushed this value toward zero, proving that standard ensembles often mistake model poor-fit for data noise.
  • RMSE vs. NLL: While the models were trained using NLL to capture variance, the BO process optimized for RMSE on a validation set. This ensured that the search for diversity didn't come at the cost of raw predictive accuracy.

Key Results

  • Uncertainty Reduction: In a noise-free environment, BODE estimated total uncertainty four times lower than the manually tuned baseline ensemble.
  • Accuracy Boost: BODE achieved an average RMSE of 0.77 Pa·s and an $R^2$ of 0.997, significantly outperforming the baseline ensemble (RMSE of 3.83 Pa·s, $R^2$ of 0.958).
  • Robustness to Noise: When 5% and 10% noise were added, BODE’s aleatoric uncertainty estimates aligned almost perfectly with the injected noise levels, demonstrating its reliability for real-world, "messy" data.

Limitations and Future Work

  • Gaussian Assumption: The current DE framework assumes that data noise follows a Gaussian distribution. While common in engineering, this may not hold for all physical phenomena.
  • Computational Cost: Running 75 iterations of Bayesian Optimization for each of the 20 ensemble members is computationally expensive, though the authors argue this is a necessary "one-time cost" for safety-critical applications.
  • Future Directions: The authors suggest testing BODE on experimental data (where noise is unknown) and exploring its application in Generative AI or more complex neural architectures beyond DCNNs.

Summary updated

BibTeX
@article{abulawi2025bayesian,
  author = {Zaid Abulawi and Rui Hu and Prasanna Balaprakash and Yang Liu},
  title = {{Bayesian Optimized Deep Ensemble for Uncertainty Quantification of Deep Neural Networks: a System Safety Case Study on Sodium Fast Reactor Thermal Stratification Modeling}},
  journal = {Reliab. Eng. Syst. Saf.},
  year = {2025},
  doi = {10.1016/j.ress.2025.111353},
}
Figure