AutoNCP: Automated pipelines for accurate confidence intervals

Yao Zhang, William R. Zame, Mihaela van der SchaararXiv 2020 · arXiv

Introduction

In high-stakes domains such as medicine and finance, point predictions are often insufficient. Decision-makers require a robust understanding of uncertainty—knowing not just a predicted risk, but the confidence interval surrounding it. Conformal Prediction (CP) has emerged as a powerful framework for this task because it provides a "frequentist coverage guarantee": if a 90% coverage rate is requested, the true value is mathematically guaranteed to fall within the predicted interval 90% of the time, regardless of the underlying data distribution.

However, the practical utility of CP is often hampered by the complexity of pipeline design. Selecting the wrong machine learning model, calibration method, or nonconformity score can result in "overly conservative" intervals that are too wide to be informative. AutoCP addresses this by automating the optimization of the entire CP pipeline to produce the narrowest possible intervals while maintaining guaranteed coverage.

The AutoCP Framework

AutoCP frames the construction of prediction intervals as a structured pipeline optimization problem. As illustrated in Figure 1, an AutoCP pipeline consists of four primary components:

  1. Machine Learning Model: Options include Ridge Regression, Random Forests, and Neural Networks.
  2. Estimator: Such as Mean, Quantile, or Density estimators.
  3. Calibration Method: Including Split Conformal, K-fold cross-validation, or Bootstrap.
  4. Nonconformity Score: The metric measuring how "unusual" a new data point is relative to the training set.
The AutoCP pipeline architecture, illustrating the flow from data subsets to final prediction intervals via an optimized selection of models, estimators, and calibration methods.

To navigate this high-dimensional search space efficiently, AutoCP employs a specialized Bayesian Optimization (BO) strategy. The authors introduce a "Divide and Conquer" approach, modeling the performance of different model classes with separate Gaussian Processes. Furthermore, a Sparse Additive Kernel Decomposition allows the optimizer to learn the effects of calibration methods and estimators across different models, significantly improving sample efficiency.

Experimental Evaluation

The researchers evaluated AutoCP across eight standard regression datasets (e.g., Boston Housing, Wine Quality, and Medical Expenditure Panel Surveys) and two causal inference tasks (IHDP and LBIDD).

The results demonstrate that AutoCP consistently achieves superior "tightness"—producing intervals that are significantly shorter than state-of-the-art benchmarks while hitting the target coverage rate. As shown in Figure 2, AutoCP (represented by the red line) maintains the lowest normalized interval length across nearly all tested datasets.

Comparison of normalized interval lengths across various datasets. AutoCP (red line) consistently produces the narrowest intervals compared to standard conformal prediction benchmarks.

Key performance highlights include:

  • Regression: AutoCP achieved interval lengths 5% to 38% shorter than the best-performing benchmarks.
  • Causal Inference: In estimating Conditional Average Treatment Effects (CATE), AutoCP provided much narrower intervals than established methods like Causal Random Forests (CRF). On the LBIDD dataset, intervals were nearly half the length of the next best competitor.

Key Insights

A central finding of this work is that the best model for point-prediction accuracy is not necessarily the best model for uncertainty quantification. AutoCP often selects different configurations than standard AutoML tools like Auto-Sklearn. By treating "tightness" (interval length) as the primary objective function, AutoCP shifts the focus from mere validity to practical utility.

Furthermore, "Source of Gain" analysis confirms that global optimization of the entire pipeline (Model + Estimator + Calibration) yields significantly better results than optimizing components in isolation.

Limitations and Future Work

While AutoCP offers a significant leap in automated uncertainty quantification, it relies on the "exchangeability" assumption common to conformal prediction, which may not hold under significant distribution shifts. Additionally, the computational cost of running a full AutoML pipeline for uncertainty is higher than simple point-prediction. Future research will explore extending this framework to classification tasks and handling non-stationary data.

Summary updated

BibTeX
@misc{zhang2020autoncp,
  author = {Yao Zhang and William R. Zame and Mihaela van der Schaar},
  title = {{AutoNCP: Automated pipelines for accurate confidence intervals}},
  year = {2020},
  eprint = {2006.14099},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.14099},
}
Figure