Optimal Ensemble Construction for Multi-Study Prediction with Applications to COVID-19 Excess Mortality Estimation

Gabriel Loewinger, Rolando Acosta Nunez, Rahul Mazumder, Giovanni ParmigianiarXiv 2021 · arXiv

Introduction

In the biomedical and social sciences, researchers frequently encounter multiple datasets—or "studies"—that address the same prediction task but originate from diverse sources. These datasets are often characterized by significant heterogeneity, arising from differences in geographic locations, data collection protocols, or population demographics. Traditional approaches to multi-study prediction typically rely on either data pooling (merging all datasets) or Multi-Study Stacking (MSS), a two-stage process that trains individual models before weighting them.

This paper introduces Optimal Ensemble Construction (OEC), a framework that moves beyond these paradigms by jointly optimizing individual study models and their ensemble weights in a single, unified step. This integrated approach significantly improves prediction accuracy, particularly in high-stakes applications like estimating COVID-19 excess mortality.

The Challenge of Heterogeneity

Standard multi-study methods often struggle with the trade-off between global trends and local nuances:

  1. Pooling (Train-on-Merged): By treating all data as a single block, pooling often ignores the unique characteristics of individual studies, leading to poor performance when study-specific effects are strong.
  2. Multi-Study Stacking (MSS): While MSS accounts for study differences by training separate models, it does so in a "decoupled" manner. Individual models are trained without the knowledge that they will eventually be part of an ensemble, resulting in sub-optimal coordination and reduced generalization to new or data-sparse domains.

Methodology: Unified Optimization

The core innovation of OEC is its "all-in-one" optimization formulation. Rather than treating model training and ensemble weighting as sequential tasks, OEC combines them into a single objective function.

The Unified Loss Function

The OEC objective function is a convex combination of two components: the sum of study-specific model losses (e.g., Mean Squared Error) and the stacking regression loss. This relationship is governed by a hyperparameter, $\eta$, which controls the trade-off:

  • High $\eta$: Prioritizes the overall performance of the ensemble.
  • Low $\eta$: Focuses on the fit of individual study-specific models.

Mathematically, the authors demonstrate that OEC serves as a "superset" of existing methods. Depending on the value of $\eta$, the framework can simplify to either standard pooling or traditional two-stage stacking, providing a flexible spectrum for model tuning.

Variants and Optimization

The framework includes two primary variants:

  • Generalist OEC: Designed for predicting outcomes in entirely new studies (zero-shot generalization).
  • Specialist OEC: Tailored for scenarios where a small amount of data is available from a specific target study, allowing the model to "specialize."

Because the joint loss function contains bilinear terms (where weights and model parameters are multiplied), the optimization problem is non-convex. To solve this efficiently, the authors employ a block coordinate descent algorithm that iteratively updates model parameters and ensemble weights, typically converging in seconds for standard datasets.

Experimental Evaluation

The researchers validated OEC using a combination of real-world mortality data and synthetic simulations.

COVID-19 Excess Mortality Estimation

The primary application involved the Short-term Mortality Fluctuations (STMF) dataset, containing weekly death counts for 38 countries. The goal was to predict "baseline" mortality—the expected death rate if the pandemic had not occurred—to accurately calculate excess deaths. This is particularly challenging for countries with limited historical data.

Simulation Studies

Data-driven simulations were used to test OEC against varying levels of "concept shift" (changes in the relationship between predictors and outcomes) and "covariate shift" (changes in the distribution of predictors). These tests allowed the authors to observe how the algorithm handles extreme heterogeneity.

Key Results and Insights

Superior Accuracy and Robustness

In the COVID-19 baseline mortality application, OEC consistently outperformed both country-specific models and traditional MSS. In several test years, OEC achieved a 30-40% reduction in Root Mean Squared Error (RMSE) compared to standard stacking. Furthermore, as the heterogeneity between studies increased in simulations, the performance gap between OEC and simpler methods widened, highlighting its resilience in "messy" real-world conditions.

The Power of "Borrowing"

One of the most significant findings was the effectiveness of Specialist OEC. The framework allows a target study with very little data (e.g., a single year of pre-pandemic history) to "borrow" trends and information from other countries. This collaborative learning ensures that the resulting baseline is far more accurate than one derived solely from sparse local data.

Stability

Unlike some complex transfer learning techniques that can occasionally lead to "negative transfer" (where external data degrades performance), OEC proved remarkably stable. Borrowing information from other studies almost always improved performance or, at minimum, matched the accuracy of a local model.

Limitations and Future Work

While OEC represents a significant advancement, the authors note several areas for further development:

  • Non-Convexity: Although block coordinate descent works well in practice (especially when initialized with MSS estimates), the non-convex nature of the joint optimization remains a theoretical challenge.
  • Model Complexity: The current study focuses primarily on linear models. While the framework is theoretically general, the computational implications and performance gains for non-linear models, such as Neural Networks or Random Forests, require further investigation.
  • Computational Overhead: Due to the joint optimization and the need for cross-validation to select $\eta$, OEC is more computationally intensive than two-stage stacking, though it remains efficient for most practical applications.

Conclusion

Optimal Ensemble Construction provides a robust and flexible framework for multi-study prediction. By integrating model training and ensemble weighting, OEC effectively navigates study heterogeneity, offering a powerful tool for global health monitoring and other fields where data is fragmented across diverse sources.

Summary updated

Abstract

It is increasingly common to encounter prediction tasks in the biomedical sciences for which multiple datasets are available for model training. Common approaches such as pooling datasets and applying standard statistical learning methods can result in poor out-of-study prediction performance when datasets are heterogeneous. Theoretical and applied work has shown $\textit{multi-study ensembling}$ to be a viable alternative that leverages the variability across datasets in a manner that promotes model generalizability. Multi-study ensembling uses a two-stage $\textit{stacking}$ strategy which fits study-specific models and estimates ensemble weights separately. This approach ignores, however, the ensemble properties at the model-fitting stage, potentially resulting in a loss of efficiency. We therefore propose $\textit{optimal ensemble construction}$, an $\textit{all-in-one}$ approach to multi-study stacking whereby we jointly estimate ensemble weights as well as parameters associated with each study-specific model. We prove that limiting cases of our approach yield existing methods such as multi-study stacking and pooling datasets before model fitting. We propose an efficient block coordinate descent algorithm to optimize the proposed loss function. We compare our approach to standard methods by applying it to a multi-country COVID-19 dataset for baseline mortality prediction. We show that when little data is available for a country before the onset of the pandemic, leveraging data from other countries can substantially improve prediction accuracy. Importantly, our approach outperforms multi-study stacking and other standard methods in this application. We further characterize the method's performance in simulations. Our method remains competitive with or outperforms multi-study stacking and other earlier methods across a range of between-study heterogeneity levels.
BibTeX
@misc{loewinger2021optimal,
  author = {Gabriel Loewinger and Rolando Acosta Nunez and Rahul Mazumder and Giovanni Parmigiani},
  title = {{Optimal Ensemble Construction for Multi-Study Prediction with Applications to COVID-19 Excess Mortality Estimation}},
  year = {2021},
  eprint = {2109.09164},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.09164},
}
Figure