From Stacked Predictions to Decisions: A Contextual Optimization Approach

Yanru Guo, Ruiwei Jiang, Siqian ShenMLxOR @ NeurIPS 2025 · OpenReview

Abstract

Optimization models and decision frameworks have seen increased use of the combination of multiple first-level learners using ensemble methods to improve predictive performance. In this work, we cast stacking as a contextual optimization problem, where a collection of $T$ first-level learners $\{h_t\}_{t=1}^{T}$ serve as context, a meta-learner estimates the target $Y$ conditional on this context, and decisions are selected to minimize the resulting conditional expected cost. We propose an empirical residual-based sample average approximation (ER-SAA) pipeline that incorporates the predictive uncertainty into the decision stage. Specifically, we (i) obtain out-of-sample first-level learners through cross-fitting, (ii) fit a meta-learner and estimate a possibly heteroskedastic scale map, and (iii) add back cross-fitted residuals as decision scenarios. This construction preserves stacking’s modeling flexibility and at the same time aligns learning with the conditional decision objective.
BibTeX
@inproceedings{guo2025from,
  author = {Yanru Guo and Ruiwei Jiang and Siqian Shen},
  title = {{From Stacked Predictions to Decisions: A Contextual Optimization Approach}},
  booktitle = {MLxOR @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=vyJmWcDTa0},
}

Loading related content...

Figure