Earned Decoupling: Class-Wise Teacher Annealing for Small, Imbalanced Medical Datasets

Knowledge distillation trains a small "student" network to copy the softened predictions of a larger "teacher," alongside the usual ground-truth labels. Teacher annealing is the practice of gradually shifting weight from the teacher's soft predictions to the true labels as training proceeds. Every annealing method we know of uses one schedule for all classes. Medical datasets strain this choice in two ways. They are skewed, with a common benign class vastly outnumbering rare pathologies. They are also tiny, sometimes under a hundred patients. The obvious fix is to anneal rare classes more slowly because the student has seen fewer of them. We think this fix is under-specified and can point the wrong way: on rare classes the teacher is often the least trustworthy supervisor in the room.

We propose that each class should decouple from the teacher only when the student has earned it. The evidence is measured on held-out folds: the student's held-out recall on class $c$ must catch up to the teacher's held-out recall on $c$. The teacher weight on class $c$ should also never fall below a floor set by how reliable the teacher proved to be on that class. The ingredient that makes this workable at small sample sizes is cross-fitting, meaning each training image receives soft labels from a teacher that never trained on it. Without it, a teacher fine-tuned on 300 images emits near one-hot predictions on those same images. Distillation then collapses into ordinary label training, and no schedule matters.

Core Claim

The claim has two parts, and both can be falsified. The first is that, on small imbalanced medical datasets, a per-class schedule driven by measured teacher and student reliability beats a per-class schedule driven by class frequency. The second is that this happens because the frequency heuristic sometimes has the wrong sign. Some rare classes benefit from leaving the teacher early, because the teacher inherited head-class bias on them. The second part is what distinguishes this work from simply writing $\psi_c(t)$ as a function of class counts.

Where This Sits

Three prior works supply the machinery. Annealing Knowledge Distillation lowers the teacher's temperature from a high value toward 1 and then fine-tunes on hard labels. Continuation KD combines a decaying loss weight with an annealed hinge margin so that the student ignores small teacher errors. Both schedules are global, and both were tested only on balanced CIFAR and GLUE. We reuse their temperature and weight decay and make them class-indexed. Rethinking Soft Labels for Knowledge Distillation shows that the right amount of teacher weight changes during training. It adapts that weight per sample using the ratio of student loss to teacher loss (WSL). That ratio does not know about class prevalence: a tail-class sample with high student loss because of little exposure looks like a sample the teacher should supervise less, or more, for reasons unrelated to reliability. WSL is therefore our closest per-sample competitor, and our method is its class-level, held-out-validated counterpart.

Two other works motivate the approach by documenting teacher bias under imbalance. Learn from Balance (KRDistill) shows that standard distillation on long-tailed data passes the teacher's head-class bias to the student. It corrects this with prototype-based feature rectification and tail-specific logit penalties, but its distillation weight is static and it was never run on medical data. MDE-MIL distills a pathology vision-language model into a whole-slide aggregator with constant weight. It raises tail-class F1 on Camelyon+-LT from 0.00% for CLAM and DSMIL to 28.57%. CLCP-MT is the nearest class-adaptive teacher-student method: it lowers pseudo-label confidence thresholds for rare classes in a Mean Teacher setup. It adapts which unlabeled samples to trust, not how much soft supervision each labeled class receives over time, and it depends on large unlabeled pools that tiny cohorts lack.

Two further works constrain the design. Unveiling COVID-19 from Chest X-ray shows that small cohorts invite shortcut learning, where models cluster images by source hospital rather than pathology. It also finds that deeper networks overfit worse than ResNet-18. This is why our students are shallow and why the teacher weight has a floor rather than annealing to zero, since ending on pure hard labels is exactly the regime that memorizes confounders. Imbalance-aware loss functions reports that metric losses alone collapse classes and must be anchored with cross-entropy. Our hard-label term is therefore always a logit-adjusted cross-entropy and never a pure metric loss.

Research Program

The method has three components.

Cross-fitted teacher labels. The first is honest soft labels. We split the training set into $k$ stratified folds, for example $k=5$, and fine-tune the teacher $k$ times, each time on $k-1$ folds. This produces out-of-fold teacher probabilities $q(x)$ for every training image. The same pass yields each class's held-out teacher recall $a^T_c$. The teacher is an ImageNet-pretrained ResNet-50 or ViT-B, fine-tuned with logit-adjusted cross-entropy. At these data sizes the $k$ fine-tunes are cheap.

The training objective. The second component is the student objective. With class $y_i$ for image $x_i$, a per-class teacher weight $\lambda_c(t)$ at training step $t$, and temperature $\tau$, the student $p_S$ minimizes

$$\mathcal{L}(t)=\sum_i \Big[(1-\lambda_{y_i}(t))\,\mathrm{CE}_{\text{adj}}(y_i, p_S(x_i)) + \lambda_{y_i}(t)\,\tau^2\,\mathrm{KL}\big(q_\tau(x_i)\,\|\,p_{S,\tau}(x_i)\big)\Big].$$

Here $\mathrm{CE}_{\text{adj}}$ is cross-entropy with frequency-based logit adjustment, and the subscript $\tau$ means softmax at temperature $\tau$. Temperature follows the Annealing-KD decay.

The update rule. The third component is the schedule for $\lambda_c$. Students are also trained in $k$ cross-fitted copies. Every $E$ epochs, we pool each copy's held-out recall into a per-class student estimate $a^S_c(t)$. When $a^S_c(t)$ reaches $a^T_c$ within a confidence margin, $\lambda_c$ decays by a fixed step; otherwise it holds. Recall is measured on held-out images because in-sample recall saturates on small data and would trigger decoupling immediately. The weight never falls below the floor $\lambda^{\min}_c = \beta \cdot a^T_c$, with $\beta$ shared across classes. A class the teacher handles well keeps some soft supervision as a regularizer, while a class the teacher handles badly can be released almost fully. The final model is the ensemble of the $k$ students, or a single student retrained on all data using the logged per-class schedule. We report both.

Analysis. Beyond headline metrics, we log every class's trajectory $\lambda_c(t)$ alongside its frequency. The central diagnostic is the correlation between decoupling time and class frequency. If rare classes always decouple last, the frequency heuristic is vindicated and our added machinery is unnecessary.

Baselines

The naive reference is a ResNet-18 student trained with logit-adjusted cross-entropy and class-balanced sampling, with no teacher. It shows whether distillation helps at all at these sample sizes. Alongside it we report the fine-tuned teacher itself, to answer whether a small student is even needed.

The strongest existing method on this question is KRDistill (Learn from Balance). It is the only distillation method in the file designed for long-tailed transfer. It reports 52.7% top-1 accuracy on CIFAR100-LT at imbalance ratio 100. We first reproduce that figure as a sanity check that our implementation is faithful. We then run it on the medical splits using the same cross-fitted teacher, since giving it in-sample soft labels would be unfair to it. Beating it means higher balanced accuracy and macro-F1 on the medical splits, with non-overlapping bootstrap intervals across seeds.

Global annealing is represented by Annealing-KD and Continuation KD. Their reported gains are CIFAR-100 ResNet-8 at 63.1% versus 61.41% for vanilla KD, and CIFAR-10 ResNet-8 at 90.21% versus 88.45%, both on balanced data. We give them the same logit-adjusted hard-label term and the same cross-fitted soft labels, so that the only difference from our method is the schedule. Per-sample adaptivity is represented by WSL under the same settings. Static KD with fixed weight and temperature is included as the conventional choice.

Two ablations isolate the key ingredient. The first, the frequency-keyed ablation, replaces the reliability-driven rule with a schedule whose decay step for class $c$ scales with $n_c / n_{\max}$, where $n_c$ is the class count. This is the obvious idea, and it has the same number of tuned knobs as ours. The second removes cross-fitting and uses in-sample teacher labels, to show whether honest soft labels are what makes any schedule matter.

Fairness is enforced in three ways. Every method gets the same backbone, augmentation, number of epochs, and an equal random-search budget, for example 30 trials over its own hyperparameters, tuned on the same inner validation folds. Cross-fitted ensembling is applied to all distillation methods or to none.

Why not just…?

Why not just use class-balanced sampling with a global annealing schedule? Balanced sampling equalizes how often the student sees each class. That removes the "rare classes got fewer updates" argument behind frequency-keyed schedules, and it is the most serious threat to any class-wise schedule. It does not equalize how reliable the teacher is per class. Our claim rests on that difference, and balanced sampling is applied inside every distillation baseline so the comparison tests exactly this. If global annealing with balanced sampling matches our method, the conclusion is that per-class schedules are unnecessary, and we will report it as such.

Why not just slow the schedule for rare classes in proportion to their frequency? This is the obvious idea, and it is our main ablation. The teacher's held-out recall on tail classes is itself the product of imbalanced fine-tuning. Learn from Balance documents that such teachers are biased precisely on the tail. Holding a rare class on a biased teacher longer would then entrench the error. The trajectory analysis settles this: we check whether any rare class decouples early under the reliability rule, and whether those classes are the ones where the frequency-keyed schedule loses recall.

Why not just use WSL's per-sample weighting? WSL already adapts teacher weight dynamically, per sample, from a loss ratio (Rethinking Soft Labels). With 5 to 20 examples in a tail class, however, per-sample training losses are noisy and measured in-sample, which is the regime where they mislead. Our rule pools evidence at the class level on held-out folds. If WSL with cross-fitted labels matches us, the lesson is that cross-fitting rather than class-level pooling is the useful part, and the in-sample-label ablation will show this.

Why not just fine-tune the large pretrained model directly and skip distillation? Deployment may tolerate a larger model, so this is a fair question. The shallow-versus-deep overfitting result and the CORDA figures in Unveiling COVID-19 from Chest X-ray suggest large models fine-tuned on small cohorts generalize poorly externally: with RSNA pre-training, the paper reports sensitivity 0.54 and specificity 0.80 on CORDA. That is evidence about fine-tuning, not about distillation, so we test it directly. The fine-tuned teacher is reported on every split, including external evaluation. If it wins on balanced accuracy, our contribution narrows to compression.

Proposed Experiments

Medical splits. The main testbed is ISIC2018 (seven dermoscopy classes with rare dermatofibroma and vascular lesions), subsampled to 100, 300, and 1,000 training images while preserving its imbalance. The test set stays full size. This gives a controlled sweep over the small-sample axis. The Pneumonia chest X-ray task used by CARE adds a second modality with a clinical minority class. CARE's reported minority recall gains (7.4% to 31.1% on pneumonia; 52.2% to 73.9% on ISIC2018) serve as context rather than direct baselines, since CARE uses bounding-box supervision that we do not assume. CORDA, with 447 CXRs and the released EIDOSLAB code, provides a small cohort for testing whether the teacher floor reduces shortcut learning. We evaluate on held-out sources and compare against the reported 0.54 sensitivity and 0.80 specificity under RSNA pre-training. HAM10000, whose majority class is about 67% of the data, serves as a second dermatology set. No reported result was found on Lacuna for any distillation method on subsampled ISIC2018 or HAM10000, so every medical-split number will be new.

Metrics. We report balanced accuracy, macro-F1, per-class recall on the two rarest classes, and expected calibration error. Results are averaged over at least ten seeds with bootstrap confidence intervals, because variance at 100 images will be large.

Decisive checks. Three results decide the claim. First, the reliability-driven schedule must beat the frequency-keyed ablation on rare-class recall at the 100- and 300-image scales. Second, the logged trajectories must show at least some rare classes decoupling earlier than frequency would predict, with those classes accounting for the gain. Third, removing the floor must hurt external CORDA performance more than internal performance, which would tie the floor to shortcut resistance rather than to fitting. If the first check fails while the ablation without cross-fitting also collapses, we will report the finding as "honest soft labels matter and schedules do not."