Compactness Losses, Hidden Strata, and Shortcuts in Lightweight Medical CNNs

Many image classifiers are trained with an extra loss that pulls every example of a class toward one shared "center" in the network's feature space. The goal is tight, well-separated clusters, which helps face and speaker verification. Medical classes are rarely one thing, though. "Pneumothorax" in a chest X-ray archive is a mix of treated patients, who already have a visible chest drain, and untreated patients, who have only the subtle lung signs. Models already lean on the drain. On one archive, accuracy measured as ROC AUC is 0.94 when a drain is present and 0.77 when it is absent (Oakden‑Rayner et al.). A loss that forces one class into one tight cluster can push in two directions. It could make the network find the feature that treated and untreated cases share, which is the pathology. Or it could make the network place untreated cases wherever the drain-defined majority sits, and misclassify the ones it cannot place.

This note proposes to settle which outcome happens and when. The main prediction is that the answer depends on model capacity and architecture. Models built from depthwise separable convolutions, the factorized, low-parameter layers used for on-device medical screening, are where single-center compactness should hurt most. We also test a simple fix. Compactness is enforced toward several centers per class, found without labels, so hidden subgroups survive instead of being averaged away. We compare this fix against the strongest annotation-free alternative, which flattens the feature spectrum. If that alternative matches it, we will say so.

Core Claim

Single-center compactness losses do not reliably improve shortcut robustness. Their effect on the worst-performing patient subgroup follows a measurable quantity: how easily the true pathology feature can be decoded linearly from the backbone's features, compared with how easily the shortcut can. When the shortcut is easier to decode, which we expect in small depthwise separable networks under strong artifact correlation, single-center compactness makes worst-group performance worse. When the pathology is easy to decode, compactness helps. Compactness toward several centers per class should remove the harm in the first regime without losing the benefit in the second.

What is new here is a controlled, dose-response test of this interaction across architectures, carried out on medical shortcut benchmarks.

Why This Gap Is Real

The compactness literature motivates the question. Softmax cross-entropy stops shaping features once classes are linearly separable, so within-class scatter is left unconstrained (Taha et al.; Zhang et al.). Several methods tighten clusters to fix this. Center loss pulls each feature toward a running class mean (Ku). Contrastive-center loss adds a push away from other class centers (Qi et al.). The Gaussian class-conditional simplex loss, or GCCS, fixes class targets at the corners of an equal-distance simplex (Ali et al.). All of these were evaluated on clean natural images and face data, never with a planted shortcut. The file itself notes that a single center can crush real sub-phenotype variation.

The shortcut literature suggests why this might matter. Ghanooni and Wagner show that contrastive features under spurious correlation suffer spectral collapse: a few dominant feature directions, meaning eigenvalues of the feature covariance, encode the easy shortcut. Flattening the spectrum restores worst-group accuracy (Spectral Dynamics). Pulling a class tightly together is a strong push toward low within-class variance, which looks a lot like that collapse. No paper in the file tests the two together.

The architecture literature constrains how we run the study. Xu et al. find that prototype regularization tuned for convolutional networks becomes unstable on vision transformers because the feature geometry differs (Xu et al.). A compactness result from one backbone therefore cannot be assumed to carry over to another, and the study has to vary the backbone.

Depthwise separable convolutions split a standard convolution into two steps: one small spatial filter applied to each channel, then a 1×1 layer that mixes channels. This cuts parameters by roughly $1/N + 1/K^2$, where $N$ is the output width and $K$ the kernel size (Xception). They are the backbone of edge diagnostic models such as Alzheimer's detection. Their behavior under shortcuts has not been studied.

The closest existing work is SpurCon. It pulls same-patient scans taken with and without devices together, using a weighted supervised contrastive loss. That is compactness used deliberately against shortcuts. SpurCon needs Patient ID metadata, and it trains only a small projection head on a frozen BiomedCLIP encoder. We train backbones end to end, use no group or patient metadata during training, and treat architecture as the experimental variable. The second-closest work is GDWS, which converts trained standard convolutions into generalized depthwise separable layers after training while keeping adversarial robustness. It says nothing about shortcuts or feature geometry. We reuse its conversion as one arm of the architecture comparison.

Research Program

The first component is a controlled testbed. We take chest X-rays from CheXpert (Irvin et al.) and paint in a synthetic tube-like line artifact. The probability that the artifact appears given the label is set to values between 0.5 (no correlation) and 0.95. At test time the correlation is reversed, which gives a clean worst group. This lets us plot worst-group accuracy against shortcut strength rather than report one operating point.

The second component varies architecture while holding parameter count and data fixed. There are four arms. The first is a standard-convolution ResNet. The second is the same topology with depthwise separable blocks, at matched FLOPs and also widened to matched parameters, so that factorization is separated from capacity. The third is the standard network converted after training with GDWS. The fourth is a small vision transformer.

The third component crosses these architectures with five training losses: plain cross-entropy, center loss, contrastive-center loss, GCCS, and the proposed multi-center variant. The multi-center variant keeps $K$ centers $\{\mu_{c,k}\}$ per class $c$. Each feature $z_i$ with label $y_i$ is pulled only toward its nearest center of its own class, and away from other classes' centers:

$$\mathcal{L}_{mc} = \sum_i \frac{\min_k \lVert z_i - \mu_{y_i,k}\rVert^2}{\sum_{c\neq y_i}\min_k \lVert z_i-\mu_{c,k}\rVert^2 + \epsilon}$$

Here $\epsilon$ is a small constant for numerical stability. The ratio form is borrowed from contrastive-center loss. Centers are initialized by k-means on early-training features. Merging is blocked by re-seeding any center that loses all of its members. $K$ is chosen from within-class clustering structure on the training set, so no group labels are used.

The fourth component consists of diagnostics. These use stratum labels for evaluation only, never for training. For each model we measure the following. We compute within-class dispersion. We compute the effective rank of the feature spectrum, following Ghanooni and Wagner. We measure how well the shortcut and the pathology can each be linearly decoded from frozen features; the gap between these two is the quantity our core claim predicts from. We also measure how far apart the drain and no-drain subgroups sit within the pneumothorax class. The core claim is supported if this decodability gap, measured under plain cross-entropy, predicts the sign of the change in worst-group accuracy when compactness is added, across all architecture and shortcut-strength cells.

Baselines

The naive reference point is plain cross-entropy (ERM) on each architecture with standard augmentation. On real data this baseline is weak in ways already on record. Binary-label training gets a robust AUROC of 0.46 on SIIM-ACR (Saab et al.). Worst-group accuracy is 35.6% on CheXpert and 0% on ISIC 2020, as reported by SpurCon. Every method must beat this baseline at matched architecture.

The strongest annotation-free competitor on our exact question is spectral flattening (Spectral Dynamics). It targets the same collapse that compactness might cause. It raised SpurCIFAR-10 worst-group accuracy from 36.5% (SimCLR) to 59.7 ± 1.0%. No reported result was found on Lacuna for it on CheXpert, SIIM-ACR, or ISIC. We will run it end to end on every architecture and give it the same tuning budget as our method: an equal number of hyperparameter trials, selected on a validation split without group labels.

NeuronTune (Zheng et al.) zeroes out penultimate-layer neurons that are more active on errors, then retrains the linear head. It reports 92.2% versus 72.6% ERM worst-group accuracy on Waterbirds. No reported result was found on Lacuna for it on medical data. We apply it after training to every trained model, including ours, because it can be combined with any of the losses.

Two references use extra supervision and serve as upper bounds rather than competitors. Saab et al.'s segmentation supervision reaches a robust AUROC of 0.82 on SIIM-ACR and 0.73 on ISIC. SpurCon reaches 73.0% worst-group accuracy on CheXpert and 65.0% on ISIC, using Patient ID and a frozen BiomedCLIP. We report how much of the gap from ERM to these references our label-free method closes, and we do not claim to beat them.

The key ablation is single-center contrastive-center loss against the multi-center loss, holding everything else fixed. This isolates whether preserving subgroups is the active ingredient. Beating spectral flattening means higher worst-group accuracy, or higher drain-absent AUROC, on the depthwise separable backbones across three seeds, with no more than one point of aggregate AUROC lost.

Why not just…?

Why not just train with more data and heavier augmentation under plain cross-entropy? Scale alone has not removed the gap. Large-archive cross-entropy still drops from 0.94 to 0.77 AUROC once the drain is absent (Oakden‑Rayner et al.). Augmentations such as cropping do not remove an artifact that lies inside the lung field. The ERM arm, run with the same augmentations at every shortcut strength, measures this directly.

Why not just flatten the spectrum and skip compactness entirely? This is the real threat. If flattening matches the multi-center loss, the subgroup-preserving fix adds nothing, and the contribution narrows to the diagnostic finding that single-center compactness is harmful in identifiable regimes. That finding still matters, because compactness losses are widely used in verification-style medical retrieval. The head-to-head comparison, plus a combined arm with both losses, settles the question.

Why not just collect segmentation masks? Masks work: robust AUROC rises from 0.46 to 0.82 on SIIM-ACR (Saab et al.). But they are costly, and the authors acknowledge this. Our target is the far more common setting where only image-level labels exist. We keep the mask-supervised model as an upper bound.

Why not just use a frozen foundation model with SpurCon? SpurCon needs patient-level counterfactual pairs, and BiomedCLIP is not an edge-deployable model. The question here is what happens inside the small networks that would actually run on devices. We still report SpurCon's numbers so readers can see the cost of dropping metadata and model scale.

Proposed Experiments

The decisive experiment is the synthetic-artifact dose-response on CheXpert. We fit worst-group accuracy against artifact correlation for each architecture and loss pair, with three seeds. The prediction fails if single-center compactness improves or leaves unchanged worst-group accuracy on the depthwise separable backbones at high correlation. It also fails if the cross-entropy decodability gap does not predict the sign of the compactness effect.

The two real-artifact checks are SIIM-ACR robust AUROC, using Saab et al.'s drain-stratified evaluation, and drain-absent pneumothorax AUROC, using the CXR14 stratification of Oakden-Rayner et al. ISIC 2020 worst-group accuracy is a check outside chest imaging, and Waterbirds is a natural-image sanity check against NeuronTune's reported figures.

A secondary test addresses the tension raised by GDWS. We apply GDWS conversion to models trained with flattened-spectrum and multi-center losses, then measure whether worst-group accuracy survives as well as adversarial robustness did in the original GDWS work. Higher-rank features may resist the conversion, which would expose a cost of combining shortcut robustness with post-training efficiency.

We report aggregate AUROC, worst-group accuracy, effective rank, and Jetson-class throughput side by side. A shortcut fix that only shows up in the subgroup metric, or only at an unaffordable compute cost, should not be read as a deployment win.