When a classifier must assign labels from a deep taxonomy (disease codes, species, legal concepts), the coarse labels near the top have thousands of examples and the leaves often have a handful. The weights a network learns for a rare leaf are therefore mostly noise, and the useful prior is obvious: a rare leaf should look like its parent unless the data clearly say otherwise. The hard part is deciding, separately for each of thousands of leaves, how far to trust the data. With so few examples, held-out validation for a single rare label is too noisy to make that call. We propose to read the answer from the disagreement between two independent fine-tuning runs. If two runs push a leaf's classifier weights in the same direction, the leaf has real signal. If they push in unrelated directions, the leaf is mostly noise and should be pulled back toward its parent. The method merges the two runs one taxonomy level at a time. Each label's weights are interpolated between the average of the two runs and the already-merged weights of its parent, using the interpolation rule from Model Stock.
The result is a single deployable model. It costs two training runs, does not need tail-label validation to set its shrinkage strength, and connects three problems that the literature treats separately. The first is the long-tail problem in hierarchical classification. The second is the cost of ensembles: deep ensembles need $N$ full runs. The third is the small-sample regime, where clinical cohorts often number fewer than 100 subjects (Bria et al.) and validation estimates are unstable.
Core Claim
Model Stock observed that weights fine-tuned from a shared pre-trained point $w_0$ lie on a thin shell around a better central point. It recovers that center from only two fine-tuned models $w_1, w_2$ by computing the angle $\theta$ between their updates $w_1-w_0$ and $w_2-w_0$, then setting
$$w_H = t\,\frac{w_1+w_2}{2} + (1-t)\,w_0, \qquad t = \frac{2\cos\theta}{1+\cos\theta}.$$
On CLIP ViT-L/14 this reached 87.8% ImageNet top-1 and 74.9% average accuracy across five distribution-shift benchmarks, about 24 times cheaper than Model Soup. Model Stock was built for flat, balanced ImageNet, and it uses one anchor and one angle per layer.
Our claim has two parts. The first is that, applied per label head, $\cos\theta$ is a cheap estimate of that label's signal-to-noise ratio. In high dimensions, two independent noisy estimates of a shared mean direction have cosine close to $\|\mu\|^2/(\|\mu\|^2+\sigma^2)$, where $\mu$ is the shared mean update and $\sigma^2$ the per-run noise variance. The resulting $t$ therefore behaves like an empirical-Bayes shrinkage factor. Rare labels will show small or negative cosines; we clip $t$ to $[0,1]$. The second part is that the right anchor for a label is not $w_0$, which is meaningless for a freshly created head. It is the merged head of its parent. Merging top-down then shrinks each sparse leaf toward the best available estimate of its ancestor, and the strength of that shrinkage is set by the data separately for every node.
The closest existing work differs in specific ways. Model Stock uses a flat label space with a single anchor. HTrans and HFT-CNN inherit parent weights only as an initialization and then let fine-tuning drift freely. The research file notes that this drift overfits badly when children have fewer than five examples. The ontological attention ensemble is, despite its name, one network with level-wise attention, not a combination of trained models. None of them uses disagreement between runs to decide how much each leaf may depart from its parent.
Research Program
The procedure has three steps. First, start from a pre-trained encoder and initialize every label head from its parent's head, as HTrans does. The heads of top-level labels start from label-name embeddings. This creates a shared starting point $w_0$ for every head. Second, run two fine-tuning runs that differ in seed, data order, and augmentation. Third, merge. The encoder is merged with standard Model Stock relative to the pre-trained weights. The heads are then visited in topological order. For each head we compute $\cos\theta$ from its two updates relative to its own $w_0$. We then interpolate between the run average and the merged parent head; in a directed acyclic graph with several parents, we use the mean of the merged parents.
The heads are re-merged after the encoder because the encoder merge changes the features the heads see. We will measure how much this mismatch costs. As a remedy, we will test a brief refit, a few hundred steps on the frozen merged encoder, that adjusts only the frequent labels. Every model, ours and all baselines, is wrapped at inference in the parameter-free coherence layer of C-HMCNN. That layer enforces that a parent's score is at least its child's. Wrapping everything removes coherence as a confound, so differences come only from the weights.
The analysis component tests the mechanism directly. We will plot per-label $t$ against training frequency and taxonomy depth, and check whether low $t$ predicts high leaf error on held-out data. If $t$ does not track frequency, the core claim is wrong regardless of headline numbers. We will also test whether the two-run disagreement provides a usable uncertainty estimate for rare labels, measured by calibration error restricted to tail labels. This is the property that motivates ensembles in the first place.
Baselines
The naive reference point is one fine-tuning run of the same encoder with flat heads. It uses per-label decision thresholds tuned on validation, which Edin et al. showed is enough to let a simple Bi-GRU beat untuned complex architectures on MIMIC-III. A second single-run reference adds parent-initialized heads (HTrans) without any merge. This shows what inheritance alone buys.
The strongest hierarchy-specific competitors differ by benchmark. On EURLEX57K it is HiDEC trained with the HBM loss, which reports Macro-F1 of 28.77 against 27.91 for HiDEC with binary cross-entropy. On ETHEC it is the hyperbolic cone model of Dhall et al., which reports species-level F1 of 0.5977 against a flat 0.4578. On MIMIC-III it is the ontological attention ensemble, which reports 0.245 Macro-F1 on 3,912 codes. Edin et al. found that correcting the Macro-F1 formula roughly doubled reported scores, so we will re-run it under their corrected protocol rather than compare against the published figure. For IHCE the file gives only a 22.4% relative Macro-F1 gain, with no reported absolute number found on Lacuna; we treat it as a re-implementation target if code allows. On the 20-dataset hierarchical multi-label suite, the competitor is C-HMCNN itself. The file reports its average rank (1.25, first on 17 of 20) but no absolute area-under-precision-recall-curve values, so no reported result was found on Lacuna for per-dataset scores. We will run it ourselves.
The strongest low-overhead ensemble competitors are the following. Flat Model Stock on the same two runs is the most important, since it differs from our method only in using per-layer angles and the $w_0$ anchor. The others are a two-model prediction average, plain weight averaging of the two runs, and single-run methods: fast geometric ensembling and the ensemble of averages. A five-model deep ensemble serves as the expensive upper reference.
Fairness is enforced by giving every method the same encoder, the same data splits, and the same hyperparameter search budget. Single-run methods get twice the epochs, so total training compute is matched with our two runs. The deep ensemble is reported with its 5× compute disclosed.
Two ablations isolate the key ingredients. The first keeps per-head $t$ but anchors to $w_0$ instead of the parent, which isolates the hierarchy. The second keeps the parent anchor but uses one global $t$, which isolates the per-label noise estimate.
Beating the field means three things. On the rarest frequency bucket (labels with fewer than 10 training examples), our method must beat flat Model Stock and both ablations on Macro-F1 and tail calibration, with confidence intervals from repeated splits that exclude zero. It must match or exceed HBM's 28.77 on EURLEX57K. And it must stay within noise of the five-model ensemble at 40% of its training cost.
Why not just…?
Why not just regularize each child's weights toward its parent's with an L2 penalty and tune the strength? This is the most serious threat, because it is a single-run version of the same prior. The difference is that one penalty strength must be chosen for thousands of labels, or chosen per label by validating on labels that have three test examples. That is exactly the evaluation instability that small-sample work warns about. We will run this baseline with a global penalty and with a penalty tuned separately for each taxonomy depth, both at matched compute. If depth-tuned L2 matches our tail Macro-F1, the per-label noise estimate adds nothing, and we will report the method as a calibration tool rather than an accuracy gain.
Why not just ensemble the two runs' predictions, or train a proper deep ensemble? Averaging predictions does reduce variance, but it never moves a noisy leaf toward its parent, so a leaf that both runs got wrong in uncorrelated ways stays wrong. It also doubles inference cost on label spaces of 8,925 codes. Five independent runs cost 5× training. Model Stock's results suggest that two runs, merged geometrically, already capture most of what many runs offer. The two-model average and the five-model ensemble are both in the comparison, and the tail-bucket results will settle whether shrinkage beats averaging.
Why not just use flat Model Stock? Model Stock's anchor for a new label head is its random or embedding initialization, which carries no hierarchical information. Its single angle per layer averages away the difference between a head for an abundant chapter and a head for a two-example leaf. The anchor ablation and the global-$t$ ablation each remove one of these differences, so the experiments show directly which one matters.
Why not just tune per-label thresholds on a flat model? Edin et al. show thresholds matter a great deal, and every method here gets them. But thresholds cannot fix a leaf whose weight direction is noise. Dhall et al.'s 0.4578 versus 0.5977 species-level gap on ETHEC indicates that representation and structure, not cutoffs, limit performance at depth.
Proposed Experiments
The main benchmarks are EURLEX57K, where 83% of labels appear fewer than 50 times, and ETHEC's four-level butterfly taxonomy. On MIMIC-III we evaluate both the full 8,925-code set and the 3,912-code setting under Edin et al.'s corrected evaluation, using PLM-ICD as the encoder. We add MIMIC-IV and the 20-dataset hierarchical multi-label suite for breadth. To probe the small-sample regime directly, we subsample MIMIC-III training patients to 100, 500 and 2,000 and repeat over ten random draws. We report bucketed Macro-F1, tail calibration error, coherence violations before the C-HMCNN wrapper, and GPU-hours.
The decisive checks come in order. First, does per-label $t$ fall with label frequency, and does low $t$ predict leaf error? Second, does the parent anchor beat the $w_0$ anchor on the rarest bucket? Third, does the method beat depth-tuned L2 shrinkage at matched compute? Fourth, does the advantage grow as the training cohort shrinks, as a shrinkage account predicts? If the first or third check fails, the idea fails.
References
- Towards AI-driven Next Generation Personalized Healthcare and Well-being
- Model Stock: All We Need Is Just a Few Fine-Tuned Models
- Hierarchical Transfer Learning for Multi-label Text Classification
- HFT-CNN: Learning Hierarchical Category Structure for Multi-label Short Text Categorization
- Ontological attention ensembles for capturing semantic concepts in ICD code prediction from clinical text
- Coherent Hierarchical Multi-Label Classification Networks
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- Hierarchy-aware Biased Bound Margin Loss Function for Hierarchical Text Classification
- Hierarchical Image Classification using Entailment Cone Embeddings
- Inheritance-Guided Hierarchical Assignment for Clinical Automatic Diagnosis
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization