Adaptive and Calibrated Ensemble Learning with Dependent Tail-free Process

Jeremiah Zhe Liu, John W. Paisley, Marianthi-Anna Kioumourtzoglou, Brent A. CoullarXiv 2018 · OpenReview

Abstract

Ensemble learning is a mainstay in modern data science practice. Conventional ensemble algorithms assign to base models a set of deterministic, constant model weights that (1) do not fully account for variations in base model accuracy across subgroups, nor (2) provide uncertainty estimates for the ensemble prediction, which could result in mis-calibrated (i.e. precise but biased) predictions that could in turn negatively impact the algorithm performance in real-word applications. In this work, we present an adaptive, probabilistic approach to ensemble learning using dependent tail-free process as ensemble weight prior. Given input feature $\mathbf{x}$, our method optimally combines base models based on their predictive accuracy in the feature space $\mathbf{x} \in \mathcal{X}$, and provides interpretable uncertainty estimates both in model selection and in ensemble prediction. To encourage scalable and calibrated inference, we derive a structured variational inference algorithm that jointly minimizes the KL objective and the model's calibration score (i.e. Continuous Ranked Probability Score (CRPS)). We illustrate the utility of our method on both a synthetic nonlinear function regression task, and on the real-world application of spatio-temporal integration of particle pollution prediction models in New England.

Introduction

Ensemble learning—the practice of combining multiple base models to improve predictive performance—is a cornerstone of modern machine learning. However, standard techniques like simple averaging or stacked generalization often rely on fixed, global weights. In many real-world scenarios, such as air quality monitoring or climate modeling, different base models may exhibit varying degrees of accuracy across different regions of the feature space. A model specialized in urban traffic might fail in rural settings, yet a "one-size-fits-all" weighting scheme fails to exploit these local strengths.

Furthermore, traditional ensembles often lack reliable uncertainty quantification. They may provide a point estimate without indicating the confidence level or the source of uncertainty. This lack of calibration—where predicted probabilities do not match observed frequencies—can be detrimental in high-stakes applications.

Methodology

Dependent Tail-free Process (DTFP)

The proposed framework addresses these limitations by introducing a feature-adaptive, probabilistic approach. At its core is the Dependent Tail-free Process (DTFP), which serves as a prior for the ensemble weights. Unlike fixed weights, the DTFP defines a random measure that varies with input features $\mathbf{x}$. By structuring base models into a tree hierarchy, the DTFP allows the ensemble to learn which groups or individual models perform best in specific regions of the feature space.

Probabilistic graphical model of the proposed ensemble framework.

As shown in the graphical model (Figure 1), the ensemble prediction $f$ is conditioned on the input-dependent weights $\mu$ and the base model predictions $\hat{f}_k$. This structure enables the decomposition of uncertainty into two interpretable components: model selection uncertainty and prediction uncertainty.

Calibrated Variational Inference

Standard Variational Inference (VI) typically minimizes the Kullback-Leibler (KL) divergence, which often leads to overconfident, poorly calibrated models. To combat this, the authors incorporate a calibration-sensitive objective: the Cramer-von Mises (CvM) distance or the Continuous Ranked Probability Score (CRPS). This joint optimization ensures that the model's predictive distribution is not only accurate but also well-calibrated, meaning its confidence intervals are statistically reliable.

Experimental Evaluation

1D Nonlinear Regression

The framework was first tested on a synthetic 1D regression task characterized by a mix of global trends and local fluctuations. This benchmark highlights the model's ability to navigate regions where different base models succeed or fail. The proposed method achieved an RMSE of 0.1531, significantly outperforming the next best baseline, nonlinear stacking, which yielded an RMSE of 0.2086.

Performance of Nonlinear Stacking on the 1D regression task (RMSE=0.2086).

Boston PM2.5 Exposure

In a real-world application, the method was used to aggregate three state-of-the-art air pollution models for the greater Boston area. Using data from 43 monitoring stations for cross-validation, the proposed approach achieved an RMSE of 0.7580, a substantial improvement over Generalized Additive Models (GAM), which achieved an RMSE of 1.0771. Qualitative analysis revealed that the model successfully identified uncertainty "hotspots" in areas with sparse monitoring or high model disagreement.

Conclusion

This work demonstrates that adaptivity and calibration are essential for robust ensemble learning. By leveraging the Dependent Tail-free Process and a calibration-aware inference objective, the proposed method provides superior predictive accuracy and trustworthy uncertainty estimates. Future work will focus on automating the discovery of model hierarchies and extending the framework to handle complex temporal dynamics more explicitly.

Summary updated

Abstract

Ensemble learning is a mainstay in modern data science practice. Conventional ensemble algorithms assigns to base models a set of deterministic, constant model weights that (1) do not fully account for variations in base model accuracy across subgroups, nor (2) provide uncertainty estimates for the ensemble prediction, which could result in mis-calibrated (i.e. precise but biased) predictions that could in turn negatively impact the algorithm performance in real-word applications. In this work, we present an adaptive, probabilistic approach to ensemble learning using dependent tail-free process as ensemble weight prior. Given input feature $\mathbf{x}$, our method optimally combines base models based on their predictive accuracy in the feature space $\mathbf{x} \in \mathcal{X}$, and provides interpretable uncertainty estimates both in model selection and in ensemble prediction. To encourage scalable and calibrated inference, we derive a structured variational inference algorithm that jointly minimize KL objective and the model's calibration score (i.e. Continuous Ranked Probability Score (CRPS)). We illustrate the utility of our method on both a synthetic nonlinear function regression task, and on the real-world application of spatio-temporal integration of particle pollution prediction models in New England.
BibTeX
@misc{liu2018adaptive,
  author = {Jeremiah Zhe Liu and John W. Paisley and Marianthi-Anna Kioumourtzoglou and Brent A. Coull},
  title = {{Adaptive and Calibrated Ensemble Learning with Dependent Tail-free Process}},
  year = {2018},
  eprint = {1812.03350},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.03350},
}
Figure