Does Lossy Downsampling Push Medical Image Classifiers Toward Shortcuts?

Convolutional networks shrink an image several times on the way to a prediction. Each stride-2 step turns every 2×2 patch of activations into a single position with a fixed channel budget. A lesion that spans only a few pixels, such as a microaneurysm in a retinal photograph, can be averaged into the background before the network has decided whether it matters. Separately, medical imaging models are known to take shortcuts: they predict disease from patient age, race, or scanner signature instead of from the pathology. This note proposes to test a specific link between the two. When downsampling washes out small, geometrically complex lesion evidence, the easiest remaining signal is a coarse demographic or scanner cue, so the network learns that cue instead.

The second half of the proposal addresses a known trap. Keeping fine detail can also keep camera-specific noise that fails to transfer across hospitals. We therefore propose a downsampling block that keeps fine detail in explicitly separated frequency channels, and a training constraint that makes only those high-frequency channels ignore acquisition-style perturbations. This requires no demographic labels and no lesion annotations.

Core Claim

Two things would be new. The first is a controlled test of whether the choice of downsampling operator changes how much a model relies on demographic and scanner shortcuts, holding data, input resolution, width, and compute fixed. The second is a block that combines two parts. One is space-to-depth downsampling, which rearranges each 2×2 neighbourhood into four channel slices rather than discarding or averaging them. The other is a fixed wavelet rotation of those slices plus an invariance loss applied only to the high-frequency slices.

Our prediction is conditional. Space-to-depth should reduce shortcut reliance mainly when the true lesion is near the scale of the stride. It should reverse and hurt cross-site transfer when high-frequency content is correlated with the acquisition site. The targeted invariance loss should remove that reversal.

Where the Pieces Come From

The mechanism comes from SPD-Conv. SPD-Conv replaces strided convolution and pooling with a parameter-free space-to-depth rearrangement followed by a stride-1 convolution. It reports large small-object gains on COCO and 64.52% versus 61.68% Tiny ImageNet accuracy for ResNet-18, but tests no clinical data, domain shift, or subgroup performance. The motivation comes from Which Shortcut Cues Will DNNs Choose?. That paper shows networks prefer simple cues: on DSprites the order is Color > Scale > Shape, and memorising colour needs about 1.2K parameters against 17.6K for shape. It never asks whether the architecture's downsampling changes that ordering; we do.

The main constraint comes from KG-DG. It found that fine vessel morphology hurts cross-domain diabetic retinopathy transfer because it is sensitive to camera focus and resolution. This "Vein Paradox" is a direct warning that naive detail preservation can backfire, and it is why our block does more than preserve.

The fairness motivation comes from two papers.

  • FairerOPTH removes age and sex shortcuts in fundus diagnosis by supervising attention with 67 annotated pathology types on a standard ResNet-101.
  • The DMIST uncertainty study shows that scanner hardware produces subgroup disparities as large as race does.

Both attack bias through supervision or post-hoc triage. We ask how much of it the backbone's downsampling produces in the first place.

For measurement we reuse ShorT. It tests whether encoding a demographic attribute actually drives unfairness by scaling that attribute's gradient and correlating encoding strength with disparity.

The Block

A detail drives the design and is often missed. A stride-2 convolution can be written exactly as a space-to-depth rearrangement followed by a small convolution over the 4C rearranged channels. This follows from standard polyphase decomposition. So the real difference between SPD-Conv and a strided convolution is not that pixels are "discarded". It is how many channels survive the reduction. A fair comparison must therefore include a strided convolution widened to match. The same identity has a practical payoff: an SPD network can be initialised exactly from ImageNet-pretrained strided weights, with the extra channel phases set to zero. This addresses the transfer problem that the file notes SPD-Conv never solved.

Our block, called frequency-split space-to-depth, works as follows. Let $a, b, c, d$ be the four phase slices of a 2×2 neighbourhood, each with $C$ channels. We apply the fixed orthonormal Haar rotation

$$ \ell = \tfrac{1}{2}(a+b+c+d),\quad h_1 = \tfrac{1}{2}(a-b+c-d),\quad h_2 = \tfrac{1}{2}(a+b-c-d),\quad h_3 = \tfrac{1}{2}(a-b-c+d). $$

Here $\ell$ is the low-frequency band and $h_1, h_2, h_3$ are the horizontal, vertical, and diagonal detail bands. The rotation is lossless and parameter-free, so the capacity of the block is unchanged. The low band passes through its own projection $g_\ell$. The three detail bands pass jointly through a projection $g_h$. The two outputs are concatenated.

During training we sample a nuisance transform $T$ from a bank of acquisition-style perturbations. The bank contains mild defocus blur, resampling by factors between 1 and 1.5, sensor noise, JPEG compression, and contrast and gamma shifts. These are deliberately kept small enough that lesions of the sizes in our datasets remain visible. At every downsampling stage $s$ we add

$$ \mathcal{L}_{\text{inv}} = \sum_s \big\| g_h^{(s)}(x) - g_h^{(s)}(T(x)) \big\|_2^2 , $$

where $g_h^{(s)}(x)$ is the projected detail-band feature at stage $s$ for image $x$. The total loss is the task loss plus $\lambda \mathcal{L}_{\text{inv}}$, with $\lambda$ tuned on validation data from the training site only.

The intent is to leave the low band free to carry global anatomy while pushing the detail pathway to encode structure that survives changes of camera. Lesions should survive; site-specific texture should not. The only supervision needed is the perturbation bank, which is why the method sidesteps the annotation burden that limits FairerOPTH.

Research Program

The work has three stages.

Stage 1: controlled synthetic testbed. We extend the WCST-ML setup from the parameter-space shortcut paper. The true label is carried by a small shape rendered at a controlled pixel size. A spurious colour or texture cue agrees with the label at a controlled rate. The tested operators are:

  • max pooling
  • strided convolution
  • strided convolution widened to 4C outputs
  • LIP learnable pooling
  • SPD-Conv
  • frequency-split space-to-depth

Every operator is trained at identical input resolution, matched FLOPs, and matched tuning budget. We measure accuracy on off-diagonal examples, where shape and cue disagree, as a function of shape size. We also measure the minimal parameter count needed to fit each cue, reusing that paper's measure, under each operator. The hypothesis predicts that the advantage of space-to-depth peaks when the shape is 2 to 8 pixels. We add a site-correlated high-frequency noise pattern as a second spurious cue. This should reproduce the Vein Paradox for plain SPD-Conv, and the invariance loss should remove it.

Stage 2: clinical transfer and subgroup disparity. We move to the clinical benchmarks described under Proposed Experiments.

Stage 3: attribution. We run ShorT on each backbone to separate two outcomes: the operator reduces the encoding of demographic attributes, or it only reduces the reliance of predictions on that encoding.

Baselines

The naive reference is a standard ResNet-50 with strided convolutions and max pooling, trained on the source domain with standard augmentation. Everything else is compared against it at matched data, epochs, and hyperparameter search budget. It anchors the question of whether downsampling matters at all.

The strongest competitor on the clinical-transfer question is KG-DG, a neuro-symbolic system that adds lesion detectors and vessel segmentation to a Vision Transformer. It reports a 5.2% cross-domain accuracy increase over existing benchmarks and 84.65% out-of-distribution accuracy for its symbolic-only configuration. The file gives no per-dataset breakdown for KG-DG, so we will rerun it on our exact source and target splits where its code allows. It uses lesion annotations that our method does not, so matching it without those labels counts as a win and exceeding it counts as a clear win.

On mammography, the reference is the DMIST study's CNN with Monte Carlo dropout triage. It reports overall Kappa of 0.646 ± 0.016 and race disparity δ at 10% rejection of 0.051 for confidence-based triage versus 0.034 for predictive variance. Beating it means lower δ at equal or better Kappa. We will also test whether our backbone combined with predictive-variance triage stacks the two gains.

For the question of whether architecture changes shortcut use, the reference is ShorT's finding of race-shortcut reliance for cardiomegaly on CheXpert (ρ = 0.469). Success is a lower ρ at equal diagnostic AUROC without using race labels in training. ShorT's own gradient-reversal model, which does use race labels, is run as a labelled upper reference.

RETFound is fine-tuned as the foundation-model reference. It reports diabetic retinopathy AUROCs of 0.943, 0.822, and 0.884 on three datasets.

Three ablations isolate the key ingredient:

  • Widened strided convolution with 4C outputs separates channel budget from rearrangement.
  • Plain SPD-Conv without the Haar split or invariance loss separates preservation from targeted invariance.
  • Invariance loss on all channels at the network output, which is ordinary augmentation-consistency training, separates band targeting from generic robustness.

The claim that band targeting matters requires frequency-split space-to-depth to beat all three on worst-subgroup and cross-site metrics.

Why not just…?

Why not just feed higher-resolution images to a standard network? Higher input resolution delays the loss of detail, but every strided stage still compresses it. It also multiplies activation memory, which is the cost the space-to-depth stem work of Sandler et al. was designed to avoid. We will run the standard ResNet at 1.5× and 2× input resolution with FLOPs matched to our model. If it closes the gap, the conclusion is that resolution, not the operator, drives the effect, and we will report that.

Why not just train a standard network with blur and noise augmentation? This is the most serious threat, because our loss is built from the same perturbations. The all-channel consistency ablation and a plain augmentation baseline test it directly. If either matches frequency-split space-to-depth on cross-site and worst-subgroup metrics, then the architectural contribution reduces to the diagnostic finding from Stage 1. The block itself would then not be a contribution, and we will say so.

Why not just use learnable pooling like LIP? LIP learns where to pool and gains 2.1 points of small-object AP on COCO Faster R-CNN, but it still collapses each window to one value. Its importance maps are learned from the same biased labels. Given that KlotskiNet predicts disease from background tiles alone, those maps may amplify spurious background texture. LIP is included in every experiment, and the Stage 1 testbed shows directly whether its importance maps lock onto the spurious cue.

Why not just use a retinal foundation model or explicit debiasing? A comparison of RETFound against traditional models finds persistent gaps between Singaporean and UK cohorts. It also finds that RETFound needs roughly 5× the fine-tuning memory, while ResNet-50 and SwinV2 reach comparable AUC on full datasets (0.914 to 0.965 versus 0.938 to 0.966). Gradient reversal needs demographic labels, and ShorT itself showed that removing age information did not fix age-related unfairness in dermatology. Both are run as references. Our block is also compatible with either, so we test the combinations as well.

Proposed Experiments

Diabetic retinopathy. We train on EyePACS and test zero-shot on APTOS 2019, Messidor-1, and Messidor-2. We report accuracy and AUROC per target site, and the gap between the source site and the worst target. Where demographic metadata exist, we stratify by it.

Ophthalmology fairness. On OculoScope we train without the 67 pathology labels and compare the sex and age mean average precision (mAP) gaps against FairerOPTH. FairerOPTH reports 92.0% mAP overall and 91.8% for females versus 90.8% for males, using those labels.

Mammography. On DMIST we report Kappa and δ across race and scanner groups.

Chest X-ray. On CheXpert we report the ShorT race correlation.

Across all clinical experiments we report worst-subgroup performance alongside the average. Every model uses ImageNet initialisation, with SPD variants built from the exact strided-weight identity. All models get three seeds and matched tuning.

Decisive checks. Three results would support the thesis:

  • On the synthetic testbed, the operator-dependent change in off-diagonal accuracy follows lesion size, and plain SPD-Conv loses to strided convolution once site-correlated high-frequency noise is added.
  • On DMIST, scanner disparity δ falls more than race disparity. That pattern would indicate the block works by neutralising acquisition differences.
  • On CheXpert, ShorT correlation falls without race labels.

The thesis is refuted if the widened strided convolution matches SPD everywhere, meaning the effect is only channel budget. It is also refuted if frequency-split space-to-depth brings no improvement over augmentation consistency on any cross-site metric.