Who Pays for Fast Sampling? Rare-Class Erasure in Few-Step Medical Image Restoration

Diffusion models restore degraded images by removing noise over many small steps, and each step is a full pass through a large network. That makes them slow. The standard fix is distillation: a "student" network is trained to jump from noise to a clean image in a handful of steps. In medicine, though, the data is skewed. Healthy tissue and common conditions vastly outnumber rare and dangerous ones. This proposal argues that these two facts interact in a predictable and clinically dangerous way. As the number of sampling steps shrinks, a restorer behaves more and more like an averaging regressor, and on skewed data the average looks like the majority class. A fast restorer may therefore make benign-looking lesions out of malignant ones, or smooth away rare pathology, while its average image quality looks fine.

We propose to measure this "tail tax" of acceleration directly, meaning the extra loss of diagnostic content on rare classes that comes from cutting steps. We also propose a fix that targets it: a few-step student that hands an input to the slow, many-step teacher only when a high-recall detector flags possible rare pathology. Rare cases are rare, so the average cost stays close to the student's few steps, but the rare cases get the full trajectory. The decisive test is whether this escalation scheme preserves minority-class sensitivity better than simply reweighting the distillation loss toward rare classes, at the same average number of network evaluations.

Core Claim

The mechanism is simple enough to state in one equation. A network trained with squared error to predict a clean image $x$ from a degraded input $y$ outputs the conditional mean. When the plausible clean images split between a majority class with prevalence $\pi_{\text{maj}}$ and a minority class with prevalence $\pi_{\text{min}}$, that mean is

$$ \mathbb{E}[x \mid y] = \pi_{\text{maj}}(y)\,\mu_{\text{maj}}(y) + \pi_{\text{min}}(y)\,\mu_{\text{min}}(y), $$

where $\mu_{\text{maj}}(y)$ and $\mu_{\text{min}}(y)$ are the class-specific posterior means and $\pi(y)$ are the class posteriors given the degraded input. When $\pi_{\text{min}}$ is small, the output is pulled toward majority morphology. A many-step stochastic sampler draws from the full posterior and can land in the minority mode. A one-to-four-step student sits somewhere between these two behaviours, so we predict that the minority fidelity gap grows as steps are removed.

This is the restoration analogue of the "feature leakage" that Training Class-Imbalanced Diffusion Model Via Overlap Optimization (DiffROP) documents for generation. DiffROP shows that on long-tailed data, tail-class samples absorb head-class features, and it places its fix mainly at late, low-noise timesteps where class-specific detail is resolved. Few-step students largely skip those intermediate timesteps. The new claims are three. First, the tail tax exists and is measurable on medical images. Second, it is concentrated on the rare classes rather than spread evenly. Third, selectively restoring the full trajectory for flagged inputs removes most of it at near-student cost.

Relation to Prior Work

Two papers supply the acceleration machinery. Efficient Diffusion-Driven Corruption Editor for Test-Time Adaptation (Decorruptor) fine-tunes an image-editing latent diffusion model to "clean" corrupted inputs before classification. It reports 47.1% ResNet-50 accuracy on ImageNet-C-bar against 29.4% for the earlier diffusion baseline, and its consistency-distilled variant runs at about 4 network evaluations, a 100× speedup. We reuse this recipe as our teacher and student. Decorruptor, however, was evaluated only on natural images with roughly balanced classes, and its single blind prompt has no notion of which fine details are pathology. Our work asks what that compression costs on a skewed medical distribution.

Super-resolution of biomedical volumes with 2D supervision (MSDSR) motivates the latency constraint. It reaches a SliceFID of 22.3 on 8× histology volume super-resolution, against 163.4 for a masked UNet. The authors say its many diffusion passes rule out intraoperative use, and that is exactly the regime where a fast restorer would be tempting.

Three papers motivate the imbalance side and constrain the design. DiffROP adds a contrastive penalty that pushes class-conditional score distributions apart. It works only at training time and still samples with many steps. On CIFAR-10LT it lowers tail-class FID from 46.51 to 39.23. FairSkin: Fair Diffusion for Skin Disease Image Generation combines resampling with a class-diversity loss and lowers the FID variance across skin-tone groups on Fitzpatrick17k from 603.74 to 327.11. It reports that resampling alone was insufficient, which warns us not to expect loss reweighting to be a complete fix. Augmenting medical image classifiers with synthetic data from latent diffusion models shows that diffusion synthesis lifts malignancy accuracy on the darkest skin tones (FST V–VI) of the Diverse Dermatology Images (DDI) set from 76.9% to 84.4%. It also finds that gains saturate at a 10:1 synthetic-to-real ratio.

All three treat the generator as an offline tool with fixed step counts. None asks what happens to rare classes when steps are cut. Rare-to-Frequent supplies a methodological hint: on natural text-to-image prompts, rare concepts benefit from treatment that varies along the trajectory. It spends extra latency on all inputs (about 72 s versus 20 s for its R2F+ variant). We instead spend extra latency only on inputs flagged as possibly rare.

Research Program

The first component is a measurement protocol. We train a teacher restorer following the Decorruptor recipe, an image-editing latent diffusion model fine-tuned on pairs of clean and corrupted images. The corruptions are modality-appropriate: blur, sensor noise, JPEG compression, low light and colour shift for dermatology photographs. The teacher is trained on the natural, imbalanced class mix. We then produce the step-count curve by sampling the teacher at 50, 20 (fast-solver), 8 and 4 steps, and by running a consistency-distilled student at 4, 2 and 1 evaluations.

For every setting, a frozen diagnostic classifier trained on clean images scores both the restored image and its clean original. We define the label-flip rate for a class as the fraction of restored images whose predicted diagnosis differs from that of the clean original. The tail tax at step count $k$ is

$$ \Delta(k) = \big[F_{\text{min}}(k) - F_{\text{min}}(K)\big] - \big[F_{\text{maj}}(k) - F_{\text{maj}}(K)\big], $$

where $F_{\text{min}}$ and $F_{\text{maj}}$ are the flip rates averaged over minority and majority classes, and $K$ is the full 50-step teacher. A positive $\Delta(k)$ that grows as $k$ falls is the core empirical claim. Beyond flips, we also check whether minority images are pulled toward majority morphology, by measuring the shift of the classifier's probability mass toward the majority class.

The second component is a pair of fixes, which are compared head to head. The first fix is tail-reweighted distillation. The student's consistency loss is weighted by inverse square-root class frequency, applied mainly at low-noise timesteps following DiffROP's timing. The second fix is selective step escalation. A lightweight detector reads the degraded input and the student's first-step estimate, and sends the input to the 20-step teacher when either the predicted probability of a minority class or the disagreement between two student samples with different noise exceeds a threshold. The threshold is set for high recall on minority classes using validation data. The costs of the two kinds of error are asymmetric: a false trigger wastes some latency, while a missed trigger risks erasing pathology. The average cost is about $4 + 20\,p_{\text{esc}}$ evaluations, where $p_{\text{esc}}$ is the escalation rate. We report $p_{\text{esc}}$ and wall-clock latency alongside accuracy.

Baselines

The naive reference point is no restoration. The diagnostic classifier is applied directly to corrupted images, which gives the accuracy floor and shows whether restoration helps at all.

The strongest existing method is the Decorruptor recipe itself, in both its 20-step solver form and its 4-evaluation consistency-distilled form, retrained on our medical corruption pairs. It is the only method in the file built for fast diffusion-based cleaning at test time. Its reported result is 47.1% ResNet-50 accuracy on ImageNet-C-bar at about 4 evaluations. No result was found on Lacuna for Decorruptor on any medical dataset, so we establish that baseline ourselves. Teacher, student and all our variants share the same backbone, training pairs, corruption set and number of GPU-hours for distillation. Our variants beat it if they lower the minority flip rate significantly below the 4-evaluation student, with paired bootstrap confidence intervals over images, while keeping mean evaluations under 8 and majority accuracy within one point.

The direct imbalance-aware competitor is a teacher trained with a DiffROP-style overlap penalty and then distilled normally. It tests whether fixing leakage at training time survives compression. We use the same hyperparameter search budget for its balance weight as for our thresholds, since DiffROP reports sensitivity to that weight.

Two ablations isolate the key ingredient. Random escalation sends inputs to the teacher at the same rate $p_{\text{esc}}$ but chooses them at random; if it matches the full method, then the detector's targeting adds nothing. Oracle escalation uses true labels to escalate exactly the minority cases, which gives the upper bound of the approach.

Why not just…?

Why not just run the teacher with a fast solver and accept 20 steps? If 20 steps already shows little tail tax and fits the latency budget, this is the honest answer, and the step-count curve will show it. The proposal matters only if the tax appears below 20 steps and deployment needs fewer. Decorruptor's move to 4 evaluations, and MSDSR's statement that its cost blocks intraoperative use, suggest that this is a realistic need. We report the full accuracy–latency frontier so that a reader can choose a point on it.

Why not just reweight the distillation loss toward rare classes? This is the most serious competitor and it is built into the design as the first fix, not dismissed. FairSkin's finding that resampling alone did not fix group disparity in generation suggests that reweighting may underperform. Restoration is a different task, though, so we compare the two fixes at matched mean evaluations. If reweighting matches escalation, the conclusion is that a cheap training change suffices, and we will say so. We also test the combination.

Why not skip restoration and train a robust, class-reweighted classifier on corrupted images? For automated diagnosis alone this may be as good, and we include it as a baseline: focal loss with corruption augmentation, applied directly to degraded inputs. Two things favour restoration even so. Clinicians look at the restored image itself, as in intraoperative histology or dermatology photographs, so whether that image preserves pathology matters independently of any classifier. And the classifier baseline gives no measure of erasure that a human reader would see. If the robust classifier beats every restoration pipeline on minority sensitivity, we report that restoration is unnecessary for automated triage.

Why not use a single-pass GAN or feed-forward network? The equation above applies directly to a feed-forward network trained with squared error, so it is the extreme form of the tail tax. MSDSR's SliceFID of 163.4 for a masked UNet against 22.3 for diffusion shows the perceptual cost. We include a feed-forward UNet restorer as the one-pass end of the step-count curve.

Proposed Experiments

The primary testbed is dermatology. We use Fitzpatrick17k, which is imbalanced by both disease and skin tone, for training and in-distribution testing, and DDI's 656 biopsy-confirmed images for held-out testing of malignancy and darker skin tones, with synthetic corruptions applied at test time. Minority groups are defined both as rare diseases and as FST V–VI. This lets us ask whether the tail tax falls on demographic minorities as well as on rare diagnoses. FairSkin's demographic parity metric (25.04 → 9.95 in its own setting) provides a fairness measure computed on restored images.

The secondary testbed is lesion-level CT. On DeepLesion, used by ISCS to test small-tumour preservation, we restore sparse-view reconstructions with a 4-step versus 50-step 2D diffusion solver. We then measure lesion detection recall stratified by lesion size and frequency of lesion type. For our purposes, small and rare lesions are the minority class. Where data access allows, we repeat the protocol on SRH volume super-resolution to connect directly to MSDSR's intraoperative motivation. Reported metrics are per-class flip rate, per-class sensitivity at fixed specificity, per-group FID, $\Delta(k)$, mean evaluations, $p_{\text{esc}}$, and wall-clock time per image.

The decisive checks are three. First, $\Delta(k)$ must be positive and increase as $k$ decreases for the 4-evaluation student on DDI and DeepLesion; if it is flat, the premise fails and we report a useful negative result. Second, at matched mean evaluations, escalation must beat both random escalation and reweighted distillation on minority flip rate. Third, the gap between escalation and oracle escalation measures how much is lost because the detector itself is trained under imbalance. A large gap would point the next step toward improving the detector's recall rather than the restorer.