Stacking is a way of combining several different predictors. A second-stage model, the combiner, learns how much to trust each base model, using the base models' predictions on held-out folds as its inputs. On large tables this reliably beats any single model. On the small cohorts typical of clinical research, often fewer than 100 subjects, it becomes fragile. The held-out predictions it learns from are noisy. The combiner overfits them. When the combiner is a logistic regression, it also sharpens probabilities and wrecks calibration. Practitioners therefore fall back on one strong model, such as a pretrained tabular foundation model, or on plain averaging. Both throw away whatever the other models know.
We propose to fix the combiner's small-sample problem by borrowing strength across datasets rather than within one. The combiner's weights are restricted to a convex mixture of the base models' predictive distributions. They are shrunk toward a prior that has been learned from how stacking weights behaved on hundreds of other tabular tasks, conditioned on cheap descriptors of the new dataset such as its size, dimensionality, class balance and the agreement among base models. At $N=30$ the weights are mostly prior. At $N=1000$ they are mostly data. Between those points the strength of shrinkage is itself learned. The claim is that this gives better log-loss and calibration than the best single model, uniform averaging, and greedy ensemble selection in the $N \in [20, 200]$ regime, at negligible extra compute.
Core Claim and Why It Is Different
Two existing lines of work shrink stacking weights, and both pool inside a single dataset. Bayesian hierarchical stacking makes weights depend on covariates and partially pools sparse input regions toward the dataset's global weights. When the whole dataset has 30 rows, the global weights are exactly what cannot be estimated, so there is nothing reliable to pool toward. Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification (cUPMI) takes a different route. It fits class-conditional Gaussians to base-model log-probabilities and samples synthetic combiner inputs to regularize an XGBoost combiner. The authors themselves report a "danger zone" below roughly 140 minority-class samples, where the covariance estimate degenerates. They also report almost no benefit for linear combiners.
Our proposal regularizes from outside the dataset entirely: the prior comes from other tasks. It needs no covariance estimate. It is aimed squarely at linear and convex combiners and at calibration, which neither paper evaluates.
Concretely, let base model $k$ produce out-of-fold predictive probability $p_k(y_i \mid x_i)$ for training row $i$, and let $z$ be a vector of dataset descriptors. The combiner solves
$$ w^\star = \arg\max_{w \in \Delta_K} \sum_{i=1}^{N} \log \sum_{k=1}^{K} w_k\, p_k(y_i \mid x_i) \;+\; \log \mathrm{Dir}\big(w;\ \alpha(z)\, m(z)\big), $$
where the symbols are defined as follows. $\Delta_K$ is the probability simplex over $K$ base models. $m(z)$ is a prior mean weight vector and $\alpha(z)$ a prior concentration, both small regression models fit on a disjoint set of meta-training tasks. They are fit by maximizing held-out log-score after applying this very objective to subsampled versions of those tasks.
The likelihood term grows linearly in $N$ while the prior term does not, so the data's influence should rise roughly in proportion to $N/\alpha(z)$. One prediction we will test is that the learned $\alpha$ is large for tiny tasks and the advantage over unregularized simplex stacking shrinks as $N$ grows. Mixing predictive distributions rather than fitting logistic coefficients on logits rules out the sharpening mechanism behind the calibration trap from the start. The prior then handles variance.
The second ingredient is the base pool. Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap stacked only tabular foundation models and found a mean pairwise Q-statistic of 0.961. It explicitly left heterogeneous pools untested. We use a deliberately mixed pool: TabPFN-v2, TabICLv2, a gradient-boosted tree model, a random forest and L2-regularized logistic regression. This follows the argument from AutoGluon-Tabular and Issues in Stacked Generalization that stacking gains come from differing inductive biases. Ting and Witten also supply two design constraints: probability inputs rather than hard labels, and a linear, low-capacity combiner. They also report that cross-validated stacking degrades on small samples, which is the failure we target.
Research Program
We build the meta-training corpus from the 153 OpenML-CC18, TALENT and TabZilla classification tasks used by Tanna et al. The tasks are split into roughly 100 meta-training and 50 meta-test tasks, with near-duplicate datasets kept on the same side.
For each meta-training task we draw repeated subsamples at $N \in \{20, 50, 100, 200, 500\}$. On each subsample we generate out-of-fold predictions for the five base models. This uses repeated 5-fold cross-validation, or leave-one-out at $N=20$; for the foundation models, refitting simply means changing the context set. We record the base models' held-out log-loss on the remaining rows of the task. From this we fit $m(z)$ and $\alpha(z)$ as low-capacity models, softmax-linear and log-linear respectively, so that the prior cannot itself overfit the corpus.
The descriptor vector $z$ contains $\log N$, $\log d$, minority-class fraction, the fraction of categorical features, and the mean out-of-fold disagreement among base models. Every descriptor is computable on the target task without labels beyond the training set.
The analysis has three parts. First, we characterize how error correlation between foundation models and tree models changes as $N$ shrinks, filling the uncharacterized cell flagged by Tanna et al. Second, we inspect the learned prior: does the optimal mixture shift toward the foundation models at small $N$ and toward trees as $N$ grows? Third, we test transfer to a real small clinical cohort that the prior never saw.
Baselines
The naive reference points are the single best base model chosen by cross-validation and uniform averaging of all five. We expect the single model to be TabICLv2 or TabPFN-v2. Tanna et al. report that TabICLv2 was statistically indistinguishable in accuracy from their best stacks: Cascade Stacking gained only +0.18% accuracy for 253× compute. On TabArena, TabICLv2 also had the lowest ECE of any method. That makes it the competitor that must be beaten on calibration, not just on discrimination.
The strongest existing ensembling competitor is greedy forward selection. Tanna et al. found it in the top accuracy clique, and it was the only ensemble that approached TabICLv2's calibration. We run it with replacement on the same out-of-fold predictions, with the same number of selection rounds as tuning budget.
Logistic-regression stacking is included as the known calibration-trap case. Bayesian hierarchical stacking is fit within each task by Markov chain Monte Carlo, with $\log N$-independent covariates chosen per Yao et al. cUPMI is applied to a logistic combiner and to an XGBoost combiner, with Ledoit-Wolf shrinkage added so that it runs at all below its danger zone.
All methods see identical out-of-fold predictions, identical splits and seeds, and the same refit base models at test time. Only the combiner differs, so base-model compute is equal by construction.
Two ablations isolate the key ingredient. The first is unregularized simplex log-score stacking, which drops the prior and tests whether the convex mixture alone fixes calibration. The second is a prior fit on label-permuted meta-training tasks, plus a constant prior that ignores $z$, which test whether the cross-task information or the mere act of shrinkage drives any gain. A pool-composition ablation, foundation models only versus the mixed pool, tests whether heterogeneity matters at small $N$.
We will count the proposal as successful if, on the 50 meta-test tasks at $N \le 100$, it beats both greedy selection and the best single model on mean log-loss. Significance is judged by a Friedman test with Nemenyi post-hoc ranks, the protocol Tanna et al. used, together with paired Wilcoxon tests. It must also show ECE no worse than the best single model. No reported small-$N$ log-loss or ECE values for these baselines were found on Lacuna; Tanna et al. report ranks, not subsampled-regime numbers. So these baseline values will be produced inside this study.
Why not just…?
Why not just use the best tabular foundation model alone? This is the most serious threat. Tanna et al. show that foundation-model ensembles barely beat TabICLv2, and the pertussis study, Multi-Task Multimodal Fusion with Tabular Foundation Models for Pertussis Booster Response Prediction, shows a frozen TabPFN-v2 outperforming XGBoost, MLP and logistic regression on cohorts of 96–158 subjects. However, Tanna et al. never tested trees alongside the foundation models, and never isolated $N<100$. If the learned prior puts nearly all mass on one foundation model and the Q-statistic between it and the tree models stays above about 0.95 at small $N$, the honest conclusion is that a single foundation model suffices. The pool ablation and the per-$N$ prior inspection are designed to show exactly that if it is true.
Why not just average uniformly, or run greedy selection? At $N=20$ these may be nearly optimal, and if the learned $m(z)$ turns out to be close to uniform, our method collapses into averaging. That outcome is a falsifiable prediction, not a hidden risk. Greedy selection fits weights on the same noisy out-of-fold predictions as any combiner, only with a discrete search. Ting and Witten's observation that cross-validation is unstable at small $N$ applies to it as well. The head-to-head comparison on log-loss across $N$ settles the question.
Why not just fit simplex weights by log-loss, or temperature-scale the logistic stack? Either would likely fix most of the calibration problem, and the unregularized simplex ablation is in the design precisely because it might. It does nothing for the variance of $K$ weights estimated from 30 rows. The prediction is that it matches our method at $N \ge 500$ but loses at $N \le 100$. If it does not lose, the cross-task prior adds nothing and we will say so.
Why not just shrink within the task, as hierarchical stacking or cUPMI do? Within-task pooling needs a reliable global estimate to pool toward. cUPMI needs a class covariance estimate, and its authors report that this degrades below about 140 minority samples. Both baselines run in the same grid, so the $N$ at which they fail is measured directly rather than assumed.
Proposed Experiments and Decisive Checks
The main experiment is the 50-task meta-test grid across the five $N$ levels. For each task, level and method we report log-loss, ECE, Brier reliability and AUC, averaged over 20 subsample seeds, with critical-difference rank plots per $N$. The decisive check is the crossover pattern. The prior should help most at $N \le 100$ and fade by $N=500$, and the learned $\alpha$ should fall with $N$. The permuted-prior ablation should erase the gain. A gain that survives permutation would mean generic shrinkage, not transferred knowledge, is responsible.
The external check is CMI-PB. Its peak-response and durability tasks have $n=158$ and $n=96$, and about 45% of subjects are missing at least one modality. The prior is applied there without refitting. Missing modalities are handled by trees that accept missing values and by per-modality TabPFN-v2 models in the pool. The reference is the reported multi-task contrastive model: AUROC 0.797 for peak response and 0.755 for durability, from the pertussis paper. Our target is to approach these AUROCs while delivering calibrated probabilities, which that paper does not report.
We will also report the pairwise Q-statistic of the mixed pool at each $N$. This directly tests whether heterogeneity lowers the 0.961 ceiling in the small-sample regime.
References
- Bayesian hierarchical stacking
- Gaussian Meta-Space Augmentation for Stacking Ensembles in Multimodal IPMN Risk Stratification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
- Issues in Stacked Generalization
- Multi-Task Multimodal Fusion with Tabular Foundation Models for Pertussis Booster Response Prediction