Reluctant Label Resolution: Letting Small Clinical Cohorts Decide How Many Classes to Predict

Small clinical studies often face a forced choice. With 270 patients, a group studying breast cancer MRI could not train a stable four-subtype classifier, so they collapsed the problem to "Luminal A versus everything else" and gave up the distinctions that matter for treatment. Ensemble research shows a related failure: adding a weaker model to a strong one often makes the combination worse than the strong model alone, because the extra component contributes more noise than signal. We argue these are the same problem. A finer class distinction that the data cannot support behaves like a weak auxiliary learner. It injects variance, and it should enter the model only when held-out evidence shows it helps.

We propose a glass-box classifier for small cohorts that grows its output granularity and its model complexity reluctantly. It starts from the coarsest clinically meaningful split. It refines a merged class into its subtypes only when cross-fitted evidence supports the refinement. At each split it admits nonlinear terms, and then frozen deep-image features, only under the same rule. The result is still a proper probability over all $K$ fine classes, so it can be scored directly against flat multiclass models. Alongside its predictions it reports a "resolution profile": which subtype distinctions this cohort can support, and which feature effects drive each one. Every effect is shown as an additive shape function, a one-feature curve that is summed into the class score, and these curves are canonicalized so that they read correctly.

Core Claim

Under extreme data scarcity, how many classes to resolve should be estimated from the data, not fixed in advance. It can be estimated with the same admission test that guards against ensemble degradation. Concretely, organize the $K$ classes into a clinically defined tree, for example Luminal A against non-Luminal-A, then Luminal B against HER2-enriched against Basal-like. At each internal node, fit a conditional additive model. Each candidate component at that node is a candidate "ensemble member": a linear term, a spline term, a residual term built on frozen deep features, or the node's split itself. A component is admitted only if

$$\hat\Delta_c - z\,\widehat{\mathrm{se}}(\hat\Delta_c) > 0,$$

where $\hat\Delta_c$ is the mean reduction in cross-fitted multiclass log loss on the finest labels when component $c$ is added, $\widehat{\mathrm{se}}$ is its standard error across patients, and $z$ is a fixed margin we calibrate on null permutations.

An unresolved node spreads its probability mass across its children in proportion to their training frequencies. The model is therefore always a full $K$-class predictor. The flat-versus-binary dichotomy becomes a continuum, and each point on it is auditable.

What is new is that one admission rule governs three choices that prior work makes separately and by hand: label granularity, nonlinearity, and whether a deep component is fused in.

Where This Sits in Prior Work

The motivation comes from two bodies of evidence.

On the ensemble side, Møller et al. found that a single domain BERT beat stacking and voting, and Zhou et al. found that averaging anomaly detectors was worse than picking the best one. In a clinical small-sample setting, Devnath et al. found that a nine-classifier majority vote over CheXNet features (92.68% accuracy on 153 X-rays) did not meaningfully beat its best single member, which reached an AUC-PR of 0.9507 on Dataset 1. All three treat degradation as something to observe after the fact. We turn it into an admission test applied during fitting.

On the label side, Deep Learning for identifying radiogenomic associations in breast cancer collapsed four subtypes to a binary task because of sample size. Its best result was AUC 0.65 from frozen GoogleNet features with a polynomial SVM, which beat fine-tuning (0.60) and training from scratch (0.54 test AUC despite 0.98 train AUC). 3D-CAM stayed binary for Parkinson's versus multiple system atrophy on 189 patients. These papers chose granularity once, by hand. We make granularity a fitted quantity.

Two papers supply the method.

Axiomatic Interpretability for Multiclass Additive Models shows that softmax additive models are non-identifiable: one can add any function to all class scores without changing the probabilities, so the raw shape functions can mislead. Its post-processing step, API, restores curves whose slopes track the probability trend. It also introduces a multiclass boosted additive model, MC-EBM. It was validated on roughly 4 million infant-mortality records, never in the hundreds-of-patients regime. We reuse API for canonicalization, but apply it within each tree node.

RAITR fits a sparse linear model first and adds splines on the residuals only when they earn it. It shows that ordinary K-fold cross-validation over-selects nonlinear terms, and it is restricted to binary treatment choices. We borrow its "linear first, nonlinear reluctantly" ordering and its warning about cross-validation. That warning is why our rule uses a standard-error margin calibrated on permutations rather than raw cross-validation gains. We extend the ordering to $K>2$ outcomes and to deep components.

One paper constrains the design. DPMIL predicts all four breast subtypes from whole-slide images (64.3% accuracy, F1 0.685, on 1,254 slides). It shows four-way resolution is achievable at roughly 1,000+ samples with an opaque model. Our claim is specifically about the regime below that, and about doing it transparently.

PECLIDES Neuro compresses Random Forest rules for binary Alzheimer's diagnosis. It shows that pruning and simplification can be nearly free in accuracy, which is consistent with our bet. However, it never asks which class distinctions are supportable.

Research Program

We will build the model in three layers.

  1. Class tree. A clinician-specified class tree supplies the candidate splits. Where no tree exists, we will also learn one by agglomerating classes with the most confusable cross-fitted predictions, and report both versions.
  2. Node models. Each node carries a conditional additive model over interpretable inputs: clinical covariates, plus a small set of named imaging descriptors where the dataset provides them. Components are added in a fixed reluctant order: linear terms, then spline terms, then a single residual term on frozen deep features.
  3. The deep residual is deliberately the last and most penalized component. It is the "strong but opaque" learner whose admission the ensemble literature tells us to distrust.
  4. Its contribution is reported as one aggregate "image residual" curve per node, not as uninterpretable per-dimension shapes.
  5. After fitting, API is applied within each node so that every displayed curve obeys the monotonicity and smoothness axioms.
  6. Admission margin. The margin $z$ is calibrated by refitting on label-permuted data. The rule should admit a component under permutation at most 5% of the time, a family-wise rate across all components.

The analysis has a second deliverable: a characterization of when the degradation boundary falls. We will sweep sample size and record which splits and components are admitted. This gives an empirical answer, per dataset, to the open question of when an auxiliary component turns from helpful variance reduction into harmful noise.

Baselines

  1. Naive references. Two reference points anchor the comparison.
  2. The first is the training-set class prior.
  3. The second is the binary collapse that practitioners actually use, scored as a four-class predictor by spreading the "rest" mass in proportion to subtype frequency. This is exactly the choice made in Deep Learning for identifying radiogenomic associations in breast cancer, whose reported number for its binary task is AUC 0.65 (95% CI 0.57–0.71).
  4. We will reproduce that frozen-GoogleNet-plus-polynomial-SVM pipeline on the same 270 patients and folds. Beating it means matching its Luminal-A-versus-rest AUC while also improving four-class log loss.
  5. Flat MC-EBM with API. The strongest interpretable competitor is the method of Axiomatic Interpretability for Multiclass Additive Models, fit flat over all $K$ classes.
  6. It receives the same inputs and the same hyperparameter budget, with its regularization tuned by nested cross-validation and at least as many trials as our margin calibration.
  7. That paper reports MC-EBM within 5% of XGBoost accuracy on large tabular data. No small-sample number for it was found on Lacuna.
  8. Beating it means a lower four-class log loss whose paired confidence interval excludes zero, at equal or fewer total shape functions.
  9. Strongest black box. For accuracy, the strongest competitor is frozen deep features plus a tuned nonlinear SVM or gradient-boosted trees, trained flat on all $K$ classes.
  10. No reported four-class small-cohort result was found on Lacuna for this setting.
  11. We do not expect to beat it. The target is to come within its confidence interval.
  12. Ablations. Two ablations isolate the key ingredient.
  13. The first keeps the tree, but forces every split to resolve. This tests whether reluctant resolution, rather than the hierarchy itself, is what helps.
  14. The second keeps reluctant resolution, but always admits the deep residual. This tests whether gating the deep component matters, which is the direct analogue of the degradation finding.

Why not just…?

Why not just fit flat MC-EBM with strong regularization? This is the real threat. Heavy smoothing might shrink unsupported class distinctions toward the prior on its own, without any explicit resolution decision. Our ablations settle it. If flat MC-EBM tuned by nested cross-validation matches our log loss and our shape fidelity on the subsampled infant-mortality experiment, the contribution reduces to the interpretive resolution profile, and we will say so. We expect a gap mainly at the smallest sample sizes, where regularization tuned for overall loss still lets rare classes such as HER2-enriched produce noisy curves.

Why not just pick the single best model by cross-validation, as Zhou et al. suggest? Global selection makes one choice for the whole problem. The breast-cancer case suggests the right answer differs by node: the coarse Luminal A split may support image features while the finer splits may not. Learning Ensembles of Structured Prediction Rules makes the analogous point that gains come from local, not global, competence. We will include global best-model selection over our own candidate set as a baseline. If it ties our method, per-node admission is unnecessary.

Why not just train K one-versus-rest binary additive models? Each rare-class model then trains on an extreme imbalance: with 270 patients, the smaller subtypes are a small fraction of the cohort. The resulting probabilities are not coherent across classes. Once they are normalized, the identifiability problem of Axiomatic Interpretability for Multiclass Additive Models returns. We run one-versus-rest additive models with softmax normalization as a further baseline and score them with the same four-class log loss.

Decisive Experiments

Experiment 1: infant-mortality subsampling. The cleanest test uses the CDC infant-mortality data from Axiomatic Interpretability for Multiclass Additive Models, where 4 million records give a near-ground-truth reference.

  • We fit MC-EBM with API on the full data to obtain reference shape functions and reference class separability.
  • We then draw 50 repeated subsamples at N = 100, 300, 1,000 and 3,000 across the 12 causes of death.
  • For each subsample we measure four things: log loss on a large held-out set, an L2 distance between learned and reference shape curves, the sign agreement of their slopes, and whether admitted splits are exactly those whose reference separability exceeds chance.
  • The prediction is that resolved splits grow monotonically with N and converge to the full tree.

Experiment 2: real small cohorts. We then apply the method to cohorts where the class question is live.

  • On the 270-patient breast DCE-MRI cohort, we predict four subtypes rather than Luminal A versus rest. This assumes the four subtype labels are recoverable from the original study's records, which we will confirm with the data holders.
  • On ADNI, we predict normal control versus mild cognitive impairment versus Alzheimer's disease. This is the three-way task PECLIDES Neuro left at binary. We subsample to the hundreds to match the target regime.
  • On BCMT, we subsample to 100–300 slides, using frozen patch features as the deep component. The full-data reference here is DPMIL's 64.3% accuracy.

Decision rule. The proposal succeeds if it meets three conditions:

  • it strictly beats binary collapse on four-class log loss;
  • it is no worse than flat MC-EBM within paired confidence intervals while showing better shape fidelity in Experiment 1;
  • the "always admit deep" ablation degrades held-out loss at the smallest N.

It fails if flat regularized MC-EBM matches it everywhere. That outcome would itself be a useful result: it would mean the binary collapse in small clinical studies was never necessary.