Spending the Ensemble Budget on the Patients Who Look Alike

Medical classifiers often look excellent when they separate patients from healthy people and then fail when they must tell one disease from another disease with similar symptoms. Ensembles are the usual remedy for brittle models. In an ensemble, several models vote, and in the version called stacking a second "meta" model learns how to weigh their votes. Combining very different kinds of model ("heterogeneous" ensembling) is supposed to help most, because models with different designs make different mistakes. The catch is that this diversity is almost always measured, and paid for, on the whole dataset. Most of that dataset consists of easy cases. This note argues that for differential diagnosis, meaning the task of picking among similar disorders, the only diversity that matters is disagreement on the small set of patients whose presentations are genuinely ambiguous between disorders. It also argues that the training compute should be allocated by that criterion.

The proposal therefore makes two contributions. The first is a measurement: we compute the error correlation between ensemble members on the confusable subset alone. We expect to find that pools which look diverse overall become nearly redundant there. The second is a procedure that works under a fixed training budget. It builds a candidate pool cheaply from checkpoints along each model family's training run. It then selects and calibrates ensemble members by their probabilistic accuracy on the confusable subset, and it decides whether the next unit of compute should buy a new model family or more checkpoints from an existing one.

Core Claim

Our hypothesis is that the gain heterogeneous ensembles give on differential diagnosis can be predicted from member disagreement on confusable cases, and not from disagreement overall. A selection rule that targets those cases should beat global ensemble selection on differential specificity and calibration at equal training compute. The motivating evidence comes from two directions. Clinical work repeatedly shows performance falling once healthy controls are removed. In a blood-plus-cognition psychiatric cohort, AUC fell from 0.89 (schizophrenia vs. controls) to 0.80 (bipolar vs. schizophrenia) (Bipolar disorder or schizophrenia multi-domain prediction). Ensemble work, for its part, shows that architectural difference does not guarantee behavioral difference. Six tabular foundation models had a pairwise Q-statistic of 0.961 across 153 tasks, meaning their errors nearly coincide (Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap). Nobody has asked whether this "diversity ceiling" is even lower on the cases that differential diagnosis depends on. If it is, globally chosen heterogeneous pools are partly wasted compute.

The closest prior system is the heterogeneous stacking pipeline of Nelson et al.. It fuses Random Forests on radiomics with ResNet-18s across 16 MRI streams through a tree meta-learner, and adds a synthetic-feature regularizer to stop that meta-learner overfitting. It reaches binary AUC 0.839 and QWK 0.595 on Cyst-X. However, it ranks risk within a single disease, trains every member independently across folds, and selects members globally. We differ on all three points: our evaluation is across disorders, our pool comes from per-family training trajectories, and selection is aimed at the confusable subset.

Research Program

The first piece is defining the confusable subset without leaking information. All base models are trained as multiclass classifiers over every disorder in the cohort, never as disease-vs-control detectors, so that differential information is available to every member. Using out-of-fold predictions, meaning each patient is scored only by models that never trained on that patient, we call a patient confusable if their top two predicted classes are both disorders and the probability gap between them is below a threshold. The threshold is chosen within inner folds. This definition follows the "common core plus specific component" picture that joint imaging analyses of the psychosis spectrum report. Wang et al. find a component shared across schizophrenia, schizoaffective and bipolar disorder alongside a temporal-lobe component unique to the first two. Under that picture, patients whose signal is dominated by the shared core are exactly the ones that models confuse.

The second piece is the candidate pool. Each model family contributes several members at roughly the cost of one training run. For neural networks, we use the cyclical learning-rate schedule of Fast Geometric Ensembling, which collects many checkpoints along low-loss paths in about one training run (Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs). We keep both cycle minima and mid-cycle checkpoints, since Zaytsev et al. found the mid-cycle checkpoints add value. For gradient-boosted trees, checkpoints at intermediate boosting rounds come free. For pretrained tabular models such as TabICLv2, members differ in context subsampling. Snapshots do not remove the need for out-of-fold predictions. Unlike Zaytsev et al., we do not trust training-set likelihood for weighting on small clinical cohorts. What snapshots do is let each fold's single run supply many members, so the k-fold multiplier is paid once per family rather than once per member.

The third piece is the combination step. Following the finding of Tanna et al. that greedy forward selection matched learned stacking while keeping calibration, we use greedy selection with replacement, followed by one temperature parameter fit in inner folds. The selection objective is a weighted log-loss that up-weights confusable patients by a factor λ. Setting λ to 1 recovers ordinary greedy selection. We deliberately avoid a learned logistic meta-learner, because the same paper reports that stacking with logistic regression had the worst log-loss rank and calibration error.

The fourth piece is budget allocation. At each step, given compute spent so far, we estimate two quantities from inner folds: the confusable-subset log-loss gain from adding a new family, and the gain from extending an existing family's trajectory. The cheaper improvement per unit of compute is taken. This turns the field's tension between heterogeneity and training cost into a question we can measure.

Baselines

The naive reference is the single best multiclass model in the pool, tuned with the same hyperparameter budget as any one family. This is the bar any ensemble must clear, and Tanna et al. found the strongest single pretrained tabular model statistically tied with three ensemble methods on general tabular data. A tie with this baseline is therefore a plausible outcome, not a strawman.

The strongest existing method is global greedy ensemble selection with temperature scaling over the same heterogeneous snapshot pool, as recommended by Tanna et al. That paper reports no single accuracy number in the file; it reports a statistical tie with stacking and the best calibration among ensembles. It is the strongest competitor because it already avoids the calibration trap, and our method differs from it only in the confusable-subset weighting λ. It uses identical members, folds, and temperature fitting. Beating it means a lower confusable-subset log-loss and higher specificity at fixed sensitivity for the target disorder against its confusable alternatives. The difference must be significant across outer folds, with overall log-loss no worse than the baseline's outer-fold standard error.

We also run learned stacking in two forms. The first is a logistic meta-learner. The second is a Random Forest meta-learner with log-space synthetic-feature augmentation as in Nelson et al., whose Cyst-X result of AUC 0.839 and QWK 0.595 is the only reported number in the file for heterogeneous clinical stacking. That result comes from a different, single-disease task, so it serves as a reference point rather than a target. Both stacking variants get the same base pool.

At equal compute, we run a homogeneous ensemble built from snapshots of the single best family alone. This is the ablation that isolates heterogeneity. A second ablation keeps heterogeneity but replaces snapshots with independently trained members, isolating the cost saving. For psychosis-spectrum classification on B-SNIP itself, we found no reported result on Lacuna; the cohort papers report component statistics such as p < 0.006 for an EEG–structure component, not classifier accuracy. On DDXPlus, the only reported numbers in the file are for language models under the hierarchical H-DDx metric (about 60% at the ICD-10 chapter level, 10–20% at subcategory level; H-DDx). We found no reported result on Lacuna for trained classifiers on DDXPlus.

Why not just…?

Why not just train one strong multiclass model that sees all disorders? This is the main threat, since much of the clinical collapse comes from training against healthy controls, and our base models already avoid that. The single-best-model baseline tests it directly. If the confusable-subset Q-statistic across families is near the 0.961 level Tanna et al. saw globally, ensembling has little to add, and we will report that the clinical fix is the training contrast, not ensembling. The proposal is only worthwhile if the measured slice diversity is substantially lower.

Why not just use ordinary greedy selection, which already works and stays calibrated? It may be enough. Whether it is reduces to whether the best λ is greater than 1, which inner cross-validation chooses and outer folds verify. If the chosen λ is consistently 1, the targeted objective adds nothing and the measurement contribution stands alone.

Why not just use a cheap homogeneous ensemble, such as snapshots or subspace training of one model? These methods cut training cost to roughly one run (Wortsman et al.; Garipov et al.). However, members sharing one training path have correlated errors, and the reported gap is visible: 90.8% for training-time stacking versus 91.9% for five independent ResNet-18s on CIFAR-10 (Zaytsev et al.). The homogeneous-snapshot ablation at equal compute measures whether the extra diversity concentrates on confusable patients, which is the specific claim here.

Why not just collect more differential cases or reweight the training loss? More data is the honest long-run answer but is not available in fixed cohorts. Up-weighting confusable patients during base-model training is a real competitor, so we include it as a baseline: each family trained with the same confusable weights, then combined by global greedy selection.

Proposed Experiments

The clinical testbed is B-SNIP, access permitting. It provides schizophrenia, schizoaffective, psychotic bipolar and control groups with EEG, gray-matter volume and resting functional MRI (Soh et al.; Wang et al.). Region-level features feed gradient-boosted trees, support vector machines, multilayer networks and a pretrained tabular model. Site is held out in the outer split, because SMGD warns that site artifacts masquerade as disease signal.

The public testbed is DDXPlus, which has many confusable pathologies and ICD-10 mappings. If simple models saturate it, we mask antecedent and history features to mimic an early presentation, and report the masking level. Without that step the confusable subset would be too small to test anything.

The metrics are confusable-subset log-loss and calibration error, pairwise differential AUC, specificity at 90% sensitivity against confusable alternatives, hierarchical F1 following H-DDx on DDXPlus, and total training GPU-hours.

Three checks decide the outcome. The first is whether the confusable-subset Q-statistic is markedly higher than the global one for the heterogeneous pool. If it is not, there is no gap to exploit. The second is whether targeted selection beats global greedy selection on the subset at equal compute and equal overall log-loss. The third is whether the budget-allocation curve shows that a new family buys more subset improvement per GPU-hour than further snapshots. If it shows the reverse, cheap homogeneous ensembles suffice for differential diagnosis and the field's push toward heterogeneity is misplaced for this task. Any of these outcomes is informative, and the second and third are what would make the method worth adopting.