Benchmarking uncertainty quantification for engineering regression under graded distribution shift

Ahmad Lawal, Rahim Pasbanigoloojeh, Asmau Lawal BilbisarXiv 2026 · arXiv

TL;DR

This paper introduces a "shift ladder" to evaluate how uncertainty quantification (UQ) methods for engineering regression handle increasingly severe distribution shifts. The study reveals a sobering reality: while modern UQ methods are highly reliable in-distribution, their performance collapses under "hard" shifts (extrapolation), and hyperparameter tuning on training data offers zero predictive benefit for out-of-distribution reliability.

What Problem Does This Paper Address?

In engineering, machine learning models are frequently deployed in conditions they never saw during training—think of a power plant operating during a record-breaking heatwave or a sensor calibrated in summer running through its first winter. Uncertainty quantification (UQ) is supposed to be the safety net, telling the engineer when the model "doesn't know."

However, most UQ benchmarks rely on random train-test splits (IID data). This assesses calibration only when the deployment looks exactly like the training data. Conformal prediction, a popular framework for providing coverage guarantees, technically only works when data is "exchangeable." This paper asks: do these UQ intervals keep their promises when we move beyond the training envelope into physically meaningful distribution shifts?

Methods

The authors introduce a three-rung shift ladder to systematically test 12 UQ methods from five different families:

  1. IID (Rung 1): The standard random split.
  2. Soft Covariate Shift (Rung 2): Training and deployment data overlap, but the deployment data is skewed toward one end of a physical variable (e.g., higher temperatures).
  3. Hard Shift (Rung 3): Either pure extrapolation (no overlap between training and test ranges) or a change of regime (e.g., a new type of engine fault).
Fig. 1: The shift ladder on CCPP (seed 0). Soft shift: 𝑃(source ∣ 𝑣) = 𝜎(−1.5 𝑧(𝑣)). Every dataset uses the same construction on its own shift variable (Table 1).

The 12 evaluated methods (detailed in the table below) include Bayesian approaches (GPs, BNNs), Ensembles, Distributional models (NGBoost), Quantile regression, and Conformal wrappers. Crucially, all models were tuned using nested cross-validation on source data only, mimicking a real-world practitioner's workflow.

Table 2 Methods, the uncertainty each represents and their settings. The last column gives the search space; the first value listed is the default, used in the fixed-setting run. Conformal methods inherit their base model’s tuned configuration (Section 4.3).

Datasets

The benchmark uses seven public engineering regression datasets where the shift variable corresponds to a real-world deployment decision, such as curing age for concrete or ship speed for naval architecture.

  • Extrapolation tasks: Concrete, Energy, CCPP, Yacht, Naval.
  • Regime changes: C-MAPSS (turbofan engine degradation) and Air Quality (field sensor deployment over time).
Table 1 Datasets, shift definitions and the grouping used for worst-group coverage.

Key Insights and Ideas

  • Marginal Coverage Hides Failure: A model might achieve 95% coverage on average, but fail miserably in the specific "worst-case" subgroups where an engineer needs it most. Under soft shifts, worst-subgroup coverage dropped 5–10% below the mean.
  • Tuning Doesn't Transfer to Hard Shift: This is perhaps the most critical finding. While tuning hyperparameters on source data improved performance for IID and soft shifts, the correlation between source-validation gains and hard-shift performance was essentially zero ($\rho = 0.04$).
  • Three Mechanisms of Failure:
    1. Trees freeze: Tree-based methods (like Quantile RF) predict a constant value and constant interval width beyond the training range, leading to 0% coverage in steep extrapolation.
  • GPs buy coverage with width: Gaussian Processes revert to their prior far from data. They maintain coverage by producing intervals so wide they become useless (sometimes twice the entire target range).
  • Ensembles under-scale: Deep ensembles correctly rank their errors (wider intervals where they are more wrong) but fail to scale the intervals enough to maintain coverage.
  • The Power of Abstention: Weighted Conformalized Quantile Regression (CQR) was the only method that "failed safely." Instead of giving a wrong interval, it recognized the lack of data and abstained (returned an infinite interval) on 85% of hard-shift points.

Key Results

  • In-Distribution: Conformal methods are rock-solid, covering at least 94.6% of test points across all datasets.
  • Hard Shift Collapse: For the 11 methods that always return an interval, mean coverage plummeted to between 0.40 and 0.75. No single family of methods consistently outperformed the others under hard shift.
  • Point Error vs. UQ: Coverage loss was highly correlated with the growth in point prediction error ($\rho = 0.74$). If the model's basic prediction fails, the UQ usually fails with it.
  • Data Splitting Matters: The authors showed that for datasets with repeated measurements (like C-MAPSS), calibrating on individual rows rather than whole "units" (engines) leads to overconfident, narrow intervals that break conformal guarantees even without a distribution shift.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study focused on univariate targets and 95% intervals.
  • Recalibration: They did not test post-hoc recalibration methods (like variance scaling) for non-conformal models.
  • Conditional Shift Detection: Current methods struggle to detect shifts that occur in the relationship between inputs and outputs ($P(Y|X)$) if the inputs themselves ($X$) don't look new.

Future Work: The authors suggest developing methods where coverage "degrades gracefully" as the effective sample size drops, rather than binary abstention. They also emphasize that for hard shifts, safety must come from physical constraints or new measurements, as source data alone cannot certify a model's behavior in the unknown.

Summary updated

Abstract

Machine learning models in engineering are often used beyond their training conditions, where reliable uncertainty quantification (UQ) matters most. Existing comparisons of regression UQ methods rely on random splits, which assess calibration only in distribution. We introduce a three-rung shift ladder (in distribution, soft shift and hard shift) built along physically meaningful shift variables, and evaluate 12 methods from five families on seven public engineering datasets: five extrapolation settings, an unseen turbofan fault mode and a winter air-quality sensor deployment. All tuning uses source data only. In distribution, conformal methods cover at least 0.946 of test points on every dataset. Under hard shift, mean coverage falls to 0.40-0.75 across the 11 methods that always return an interval, and no systematic difference in method ranks is detected (Friedman p = 0.63). Weighted conformalised quantile regression stays near nominal on the points it answers, but abstains on 85\% of hard-shift points on average and covers only 0.67 of the few Concrete points it answers. Source-validation gains predict improvement in distribution and under soft shift, but not under hard shift (Spearman's rho = 0.04). Engineering evaluations should split calibration data by deployment unit, report subgroup coverage and sharpness, and treat abstention as an explicit operational outcome.
BibTeX
@misc{lawal2026benchmarking,
  author = {Ahmad Lawal and Rahim Pasbanigoloojeh and Asmau Lawal Bilbis},
  title = {{Benchmarking uncertainty quantification for engineering regression under graded distribution shift}},
  year = {2026},
  eprint = {2610.35356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35356},
}
Figure