Spending No Labels on Ensemble Weights: Knowledge-Anchored Combination of Source Experts for Rare Clinical Cohorts

When a hospital wants a prediction model for a rare condition, it may have only a few dozen labeled patients. It will usually have far more data on related conditions. A natural move is to train one expert model per related condition and combine their predictions with a weighted average. The weighting step, however, is where small cohorts hurt most. Every existing way of learning ensemble weights fits them on held-out labels, and a few dozen labels cannot pin down a weight vector without the weights overfitting the particular patients who happened to be held out. This note proposes a different starting point. Choose a prior weighting using only information that costs no target labels: how clinically similar each source condition is to the target, read from a knowledge graph or from unlabeled target covariates. Then let the scarce target labels correct that prior only as far as they can justify.

The claim is concrete and falsifiable. Take ensemble weights that are pulled toward a similarity-based prior instead of toward uniform weights. With 10 to 100 labeled target patients, these weights should beat three alternatives: uniform averaging, shrinkage toward uniform weights, and hard selection of the most similar sources. The advantage should shrink as the target cohort grows, and it should vanish when the similarity scores are randomly permuted. If the permuted-prior control does as well as the real prior, the idea is wrong.

Why this gap is real

Two lines of work handle scarce target cohorts by borrowing from related data. They take opposite routes, and neither weights models continuously. Boosting Transfer Learning with Survival Data from Heterogeneous Domains (Transfer Survival Boosting, or TSB) reweights individual source patients by their similarity to the target and trains one boosted survival-tree model. Bridging data gaps of rare conditions in ICU: a multi-disease adaptation approach for clinical prediction (KnowRare) builds a condition knowledge graph from diagnosis co-occurrence, lab and vital-sign distributions, and shared medications. It uses that graph to hard-select the top-$k$ most similar source conditions, then trains one shared representation by adversarial domain adaptation. KnowRare also supplies the key motivating fact: naively pooling all conditions degrades rare-condition performance. Its authors call this the "data-volume paradox." It tells us that how much each source is trusted matters, and that a binary in-or-out decision is a crude way to set it. We keep KnowRare's knowledge graph as a source of prior information. We replace its hard selection and single shared model with a continuous, label-correctable weighting over separately trained experts.

On the ensembling side, the methods that learn continuous weights all assume a usable validation set. MVIToxNet, in Integrating Multiview Information for Enhanced Deep Learning-Based Acute Dermal Toxicity Prediction, learns weights over its top-$K$ models on validation scores. The wave-forecasting system in An integrated framework that combines machine learning and numerical models to improve wave-condition forecasts fits weights by ridge regression or by Exponentiated Gradient, an update that keeps weights non-negative and summing to one. Lazy Learning Meets the Recursive Least Squares Algorithm weights local models by inverse leave-one-out error. These papers motivate the proposal: they, and the forecasting comparison in A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition, show that combining beats picking a single winner. They also constrain it. MVIToxNet's learned weights gained only up to 0.73% recall and 1.11% MCC over the single best model, a margin a tiny validation split can easily erase. In addition, Unveiling COVID-19 from Chest X-ray with deep learning: a hurdles race with small data shows that small evaluation sets reward models that exploit collection artifacts. We borrow the Exponentiated Gradient update as our solver. What we change is the point it starts from and is pulled back toward.

Core mechanism

The first design choice removes the usual data-splitting problem. Each expert $f_j$, for $j = 1, \dots, M$, is trained only on source condition $j$. As a result, every labeled target patient can be used to fit the weights, and none has to be withheld from base-model training. We add one target-only expert, $f_0$, whose predictions on target patients are produced out-of-fold so that its apparent accuracy is not inflated.

Each source $j$ receives a label-free similarity score $s_j$. On the ICU tasks, this is cosine similarity in the condition knowledge-graph embedding. On survival tasks, it is a covariate-distribution similarity computed from unlabeled target records. The scores define a prior weighting $\pi_j \propto \exp(s_j/\tau)$, where the temperature $\tau$ controls how peaked the prior is. The weights $w$, constrained to the probability simplex, then solve

$$ \hat w = \arg\min_{w \in \Delta} \; \frac{1}{n}\sum_{i=1}^{n} \ell\Big(\sum_{j=0}^{M} w_j f_j(x_i),\, y_i\Big) \;+\; \lambda\, \mathrm{KL}(w \,\|\, \pi). $$

Here $n$ is the number of labeled target patients, $\ell$ is the task loss (cross-entropy, or a censoring-weighted Brier score for survival), and $\lambda$ sets how strongly the weights are pulled back to the prior. The problem is convex and solved by Exponentiated Gradient started at $\pi$. We choose $\lambda$ and $\tau$ by leave-one-out on the target labels, which is cheap because only the combination step is refit.

The mechanism's expected behavior follows from the objective. With few labels, $\hat w$ stays near $\pi$. As $n$ grows, the loss term dominates and the prior washes out. The prior should therefore help most at small $n$, which is where the claim can be tested.

Research Program

The work has three parts. First, we build the expert pools. On MIMIC-III and eICU, we rebuild KnowRare's condition knowledge graph from its three stated relation types. We then train one lightweight head per source condition on top of a shared pretrained time-series encoder, which keeps both training and inference cost close to that of a single model. On the MAGGIC heart-failure meta-database, we train one survival model per sub-study (DIAMO, DIG, ECHOS, Euro, IN-CH), treat each sub-study in turn as the target, and derive similarity from unlabeled target covariates.

Second, we run a controlled sweep over target label budgets $n \in \{10, 25, 50, 100, 200\}$. Each setting uses many random draws of the labeled subset, and evaluation is on the remaining target patients.

Third, we analyze when the prior helps. We regress per-condition gains on a measure of prior quality: the rank correlation between $\pi$ and the weights an oracle would fit on the full target cohort. The proposal predicts that gains track prior quality. That relationship tells a practitioner when the knowledge graph is worth building.

Baselines

The naive reference point is uniform averaging over all experts. It estimates no parameters, so it cannot overfit the target labels. It is the classic robust choice under tiny samples, and it is the baseline small-cohort practitioners actually use. It runs on exactly the same expert pool as our method. Beating it means higher mean target AUPRC (or C-index) at $n \le 50$, with a paired bootstrap interval across label draws that excludes zero.

Pooled training fits one model on all source data plus the target labels. It is included because KnowRare's data-volume paradox predicts that it fails, and the failure should be reproduced before the ensemble's gains can be credited to anything else. It uses the same encoder and training budget as the experts.

KnowRare itself is the strongest existing method on the ICU tasks. It uses the same knowledge graph with hard top-$k$ selection and adversarial adaptation into one model. We run it under the same target label budgets and tune $k$ and its adversarial weight with the same leave-one-out budget we give $\lambda$ and $\tau$. Its paper reports improvements of up to 17.0% in AUPRC over the best domain-adaptation baseline across its five tasks. The file gives no absolute AUPRC values, so we will re-run it rather than quote a number. Beating KnowRare means matching or exceeding its AUPRC at $n \le 50$ while using no adversarial training. Our method can also sit on top of KnowRare: its adapted model can serve as one expert in the pool.

On MAGGIC, the strongest baseline is TSB. It reports the highest C-index in 4 of 5 sub-studies, including a C-index of 0.677 on the Euro study against 0.500 for a Cox model trained only on target data (Boosting Transfer Learning with Survival Data from Heterogeneous Domains). We run TSB and our method on identical target subsets. We count a win if our method exceeds TSB's C-index on Euro and on the majority of sub-studies at matched $n$.

Two ablations isolate the key ingredient, the similarity prior. Uniform-anchored shrinkage is the same objective with $\pi$ set to uniform weights, with $\lambda$ tuned identically. Permuted-prior keeps the same objective but randomly permutes the similarity scores across sources. We also report unregularized learned weights ($\lambda = 0$) and winner-take-all selection by leave-one-out error, which together represent the MVIToxNet- and lazy-learning-style choices.

Why not just…?

Why not just average all experts uniformly? Equal weights are hard to beat when labels are scarce, and they may well beat unregularized learned weights here. But uniform averaging ignores the data-volume paradox: it gives dissimilar conditions the same say as close ones, which is the failure KnowRare documents for pooling. The $n$-sweep settles the question directly. If uniform averaging matches the knowledge-anchored weights at $n = 10$, then the prior carries no usable information at the regime that matters, and we would report that.

Why not just shrink learned weights toward uniform? This is the real threat. It captures most of the variance reduction without any knowledge graph. The uniform-anchored ablation is built to answer it: it uses the identical objective, solver and tuning budget, and changes only the anchor. If it ties our method, the contribution collapses to "regularize your weights," which is already standard, and we would abandon the claim. The permuted-prior control guards against a second failure: a non-uniform anchor helping merely because it is peaked rather than because it is correct.

Why not just use KnowRare's hard top-$k$ selection and fine-tune one model? Hard selection is a special case of our prior, with uniform weights on the top $k$ sources and zero elsewhere. It cannot use target labels to demote a similar-looking but unhelpful source, or to recover a dissimilar but useful one. KnowRare's reported gains show that selection helps. Whether continuous, label-corrected weights help further is exactly the head-to-head comparison above. Because KnowRare's adapted model can join our expert pool, the two are complementary rather than exclusive.

Why not just pool everything, or synthesize more patients? Pooling is directly contradicted by KnowRare's finding for rare conditions, and we re-verify it as a baseline. Synthetic data is contradicted in a related data-scarce setting by Limitations of Synthetic Data Generation in Specialized Data-Scarce Domains. There, no generative method consistently beat AugMix, a simple non-generative augmentation. That evidence comes from trauma images rather than EHR records, so we treat it as indicative, not decisive, and do not pursue synthetic data here.

Proposed Experiments and Decisive Checks

The primary benchmarks are rare-condition prediction on MIMIC-III and eICU (90-day mortality and 30-day readmission, following KnowRare's task definitions) and leave-one-study-out survival prediction on MAGGIC. We report AUPRC and AUROC, or C-index and integrated Brier score for survival, plus expected calibration error, averaged over at least 50 label draws per budget.

Three results would be decisive. First, knowledge-anchored weights must beat uniform-anchored shrinkage at $n \le 50$, and the gap must close by $n = 200$. Second, the permuted-prior control must lose that advantage. Third, gains across conditions must correlate positively with prior quality. If all three hold, the result is a practical rule: in small cohorts, spend knowledge rather than labels on setting ensemble weights. If the first two fail, the note's central claim is refuted, and the better advice is simply to regularize toward uniform weights.