Coarse When Uncertain: Hierarchical Prototypes for Safer Referral Under Hospital Shift

A medical image classifier trained at one hospital often fails at another, because new scanners, cameras, or patient populations change the images in ways unrelated to disease. The most dangerous failures are confident ones. A model can call an out-of-hospital melanoma a benign nevus with near-certainty, and no uncertainty filter flags it for a human. Clinical labels, however, are almost always organized as a tree: benign versus malignant above the seven dermoscopic diagnoses, or healthy versus referable retinopathy above the five retinopathy grades. A clinician who is unsure of the exact subtype still commits to the coarse call that drives triage. We propose to train classifiers that behave the same way. When the image comes from an unfamiliar hospital, the model should give up on the fine label first and keep a reliable coarse label, rather than flip across the coarse boundary with high confidence.

The mechanism comes from hierarchical regularization, which is the practice of tying the representation of each fine class to its parent class so that rare classes borrow statistical strength from common ones. Concretely, each disease class is represented by a prototype, meaning a reference point in feature space. Fine-class prototypes are constrained to be small offsets from their parent's prototype. Confidence at each level of the tree is read from distances to prototypes instead of from softmax scores. The core bet is that this tying reshapes the errors a model makes under domain shift. Mistakes that cross a coarse boundary should become rarer and lower-confidence, and mistakes should concentrate among siblings, where backing off to the parent is safe. That property can be tested directly. It is also exactly the property that cheap post-hoc fixes on a flat classifier cannot create if the flat features have already merged the coarse classes.

Core Claim

Our claim is that, with a single source hospital and no target data, prototype tying along the disease tree lowers the rate of confident coarse-boundary errors on unseen hospitals at matched coverage. Coverage here is the fraction of cases the model answers without referral. We further claim that this gain does not come from simply summing a flat model's fine probabilities into coarse ones and recalibrating. If that post-hoc route matches us, the claim is false, and the experiments are built to detect it.

Two lines of prior work motivate the claim. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift shows that standard and Bayesian networks produce low-entropy false positives on out-of-domain data (EyePACS to APTOS, Kermany to ChestX-ray14). It responds with post-hoc logit adjustments on flat binary outputs, and we keep its evaluation setting but move the fix into training and across a hierarchy. Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention and Prototype Propagation Networks show that letting fine classes inherit from ancestors rescues rare classes: PPN+ reaches 48.00% versus 33.17% for flat prototypical networks in 5-way 1-shot WS-ImageNet-Pure. Both were evaluated only on independent, identically distributed splits. Our question is whether the same inheritance also buys robustness and calibrated abstention under covariate shift.

Relation to the Closest Work

Several papers sit near this intersection, and each differs from the proposal in a specific way.

Taxonomy Adaptive Cross-Domain Adaptation (TADA-MI) handles hospitals with different label granularities by projecting gradients of new fine classes into a subspace computed from shared anchor classes. It needs 5-shot labeled target data, regularizes the parameter trajectory rather than the feature geometry, and does not study abstention. Our method uses no target labels and targets referral behaviour.

Taxonomy-Structured Domain Adaptation preserves a tree over domains while keeping the label space flat. It also proves that uniform adversarial alignment destroys such structure, which constrains our design: we deliberately avoid domain-adversarial alignment.

HCDG and HASD use "hierarchy" in an operational or spatial sense: data-level versus task-level consistency, or domain to slide to patch. Neither imposes a disease taxonomy.

Contrastive Centroid Supervision (FCCL) supplies most of our method. It pulls single-source embeddings toward class centroids and improves robustness to camera shift without target data. Its centroids, however, are flat and independent, and they become noisy for rare classes such as melanoma.

KnoBo motivates the general bet that a structural medical prior resists hospital shortcuts. It gets that prior from textbook-derived concept bottlenecks and paired text, which we do not require.

Research Program

The model is a standard image encoder followed by L2-normalized embeddings, as in FCCL. Each coarse node $p$ has a prototype $\mu_p$, and each fine class $c$ with parent $p(c)$ has a prototype

$$\mu_c = \mu_{p(c)} + \delta_c, \qquad \mathcal{L} = \sum_{\ell} \mathcal{L}^{(\ell)}_{\text{proto}} + \lambda \sum_c \lVert \delta_c \rVert^2 .$$

In this objective, $\delta_c$ is a learned residual. $\mathcal{L}^{(\ell)}_{\text{proto}}$ is a cosine-distance prototype cross-entropy computed at tree level $\ell$, either coarse or fine. $\lambda$ controls how tightly children are tied to their parent. Coarse prototypes are updated from all their descendants' samples, so a rare child's prototype inherits a well-estimated anchor and only has to learn a small offset.

At test time the model predicts top-down. It answers at the fine level if the fine-level confidence passes a threshold. Otherwise it answers at the coarse level if the coarse confidence passes its own threshold. Otherwise it refers the case to a clinician. Confidence at each level is the distance margin between the nearest and second-nearest prototype at that level. Thresholds are fit on a held-out source split only, using a split Platt-scaling step borrowed from the referral paper, so that target data never influences any design choice.

The analysis has three parts. First, we measure how errors on target hospitals distribute between within-parent and cross-parent mistakes. Second, we measure how confidence on cross-parent errors compares with confidence on correct predictions. Third, we trace how both quantities move as $\lambda$ sweeps from zero, which corresponds to a flat model with multi-level heads, to strong tying. We also test sensitivity to the tree itself by replacing the clinical tree with a random tree of the same shape. If the gain comes from clinically meaningful grouping, the random tree should erase it.

Baselines

The naive reference is empirical risk minimization: a flat seven-class (or five-grade) cross-entropy classifier with the same backbone, augmentation, and training schedule. It answers at the fine level or refers based on max-softmax confidence. Every other method must beat this to matter.

The strongest competitor on the referral question is the flat classifier combined with post-hoc hierarchical marginalization, where fine probabilities are summed into coarse ones. Thresholds come from the Split Platt Scaling procedure of Avoiding Catastrophic Referral Failures, giving the same three-way answer-fine, answer-coarse, or refer behaviour as our method. This is the most direct threat because it delivers coarse backoff at zero training cost. For this method and these splits, we found no reported result on Lacuna; the file lists no numbers for SR, SSR, or SPS.

The strongest competitor on single-source robustness is FCCL, run both with and without the same marginalization and Platt step. On EyePACS to APTOS, FCCL reports a target score of 0.767 against 0.734 for its baseline and 0.799 on source (Contrastive Centroid Supervision). We will reproduce this setting and first check that our FCCL reimplementation lands near 0.767 before any comparison.

Two ablations isolate the key ingredient. The first uses multi-level prototype heads with independent coarse and fine prototypes (no tying, $\lambda = 0$). The second uses tying but reads confidence from softmax instead of prototype distances. The first ablation separates prototype tying from simple multi-task supervision, and the second separates it from the distance-based confidence readout.

Fairness rules apply to all methods. Every method uses the same ImageNet-pretrained backbone, augmentation pipeline, number of epochs, and a hyperparameter search of equal size. Model and threshold selection use only a held-out source split, and results are averaged over five seeds with bootstrap confidence intervals.

Beating the strongest baseline means one specific outcome. At matched coverage on target hospitals, our method must show a lower rate of confident coarse-boundary errors than marginalized FCCL with Platt scaling, with non-overlapping 95% intervals, while fine-level accuracy on answered cases does not drop.

Why not just…?

Why not just sum a flat model's fine probabilities into coarse ones and recalibrate? This is the real threat, and the proposal is built around testing it. Marginalization can only fix a cross-parent error if the flat model placed substantial probability mass on the correct parent's children. The referral paper's finding that out-of-domain false positives have low entropy suggests the mass often lands confidently on the wrong side. Whether this holds for seven-class dermoscopy is an empirical question. The head-to-head against marginalized FCCL with Platt scaling settles it. If marginalization matches us, we will report that training-time tying is unnecessary for referral and present the work as an analysis result.

Why not just train a separate binary benign/malignant classifier for triage? A dedicated coarse model is cheap and is a reasonable reviewer instinct. It discards the fine labels, however, and it is still a flat classifier under shift, exposed to the same confident failures the referral paper documents on binary EyePACS to APTOS. We include this classifier as an extra coarse-only reference, judged by the same coarse confident-error metric. If it equals our method at the coarse level, our advantage reduces to delivering fine labels whenever the model is confident. That is still useful, but it is a smaller claim.

Why not just use flat centroid contrast, which already resists camera shift? FCCL is the base we build on, not a rival to dismiss. Its centroids are independent, so rare classes such as melanoma or dermatofibroma get high-variance estimates. Nothing in its geometry places siblings near a common anchor. Running FCCL with $\lambda = 0$ under the identical prototype head isolates exactly the effect of the tying.

Why not just use heavy style augmentation such as Fourier amplitude mixing? HCDG found that Fourier style mixing alone was not enough for segmentation and needed task-level consistency added. In any case, augmentation addresses low-level appearance, not how errors distribute across the label tree. Augmentation is orthogonal to our method, so we will apply the same Fourier mixing to all methods in a secondary run. If the gap between methods closes under augmentation, that is a meaningful finding.

Why not use a textbook-grounded concept bottleneck like KnoBo? KnoBo reports a 32.4% average gain over fine-tuned vision transformers on confounded splits, so it is a strong prior. It depends on retrieved medical text and vision-language grounding. Our question is whether a label tree alone, which every labeled dataset already carries, gives comparable shortcut resistance. We treat KnoBo as context rather than as a matched-compute baseline.

Proposed Experiments

The primary testbed is dermoscopy. We train on HAM10000 using the two-level benign/malignant-over-seven-diagnoses tree that TADA-MI also uses. We test on the BCN20000 and MSK cohorts from the skin-lesion UDA benchmark, which documents that adaptation methods become unstable at low melanoma prevalence. That makes it a natural stress test for a rare malignant child under shift.

The second testbed is diabetic retinopathy. We train on EyePACS and test on APTOS, using the five-grade tree with the healthy (grades 0–1) versus referable (grades 2–4) split taken from the referral paper. This setting lets us compare directly against both FCCL's reported setting and the referral paper's protocol.

The third testbed is chest X-rays, moving from Kermany pediatric images to ChestX-ray14. It probes a larger population shift, with a coarse normal-versus-abnormal parent placed over finding labels wherever the label sets align.

We report the following metrics on each testbed. The headline metric is the coarse-boundary confident-error rate at fixed coverages of 80% and 90%. The other metrics are the area under a hierarchical risk–coverage curve (with an error at the coarse level costing more than an error at the fine level), the fraction of target errors that stay within the correct parent, fine and coarse accuracy on answered cases, and recall on the rarest malignant class.

The decisive check is a single comparison: our method versus marginalized FCCL with split Platt scaling on BCN20000 and MSK, at matched coverage. Two diagnostics accompany it. In the $\lambda$ sweep, the within-parent error fraction should rise with tying strength on target data but not necessarily on source data, which would show that the effect is specific to shift. In the random-tree control, the gain should disappear. If tying lowers confident cross-parent errors only with the real clinical tree and only under shift, the claim stands. If marginalization closes the gap, we will say so plainly.