Projecting Out Scanner Variation, Not Patient Variation

Medical image classifiers trained at one hospital often fail at the next. Part of the reason is that a different camera, scanner or stain changes how the same tissue looks. Person re-identification, the task of matching the same pedestrian across different surveillance cameras, has long dealt with a structurally similar problem. Its field calls it intra-class appearance variation: two photos of one person from different cameras can look less alike than photos of two different people from the same camera. One clean answer from that field is the discriminative null space (Learning a Discriminative Null Space for Person Re-identification). It is a linear projection, computed in closed form, under which all images of one identity collapse to a single point while different identities stay apart.

This note proposes porting that idea to medical domain shift with one deliberate change in what counts as an "identity." We do not collapse every image of a disease class to one point. Instead we treat each individual case as an identity and its simulated re-acquisitions as the "cameras." We then project deep features onto the directions along which re-acquisition changes nothing, while keeping the differences between patients, including disease severity and subtype. In two sentences: estimate, from style-perturbed copies of the same training image, the feature directions that only acquisition moves, and remove exactly those directions with a closed-form projection on a frozen backbone. This targets scanner shift without erasing the clinically meaningful variation inside a diagnostic class that class-level compactness methods throw away.

Core Claim

The closest existing work collapses within-class spread around a single class centroid. Contrastive Centroid Supervision Alleviates Domain Shift in Medical Image Classification (FCCL) pulls each embedding toward a moving-average class mean and raises EyePACS target accuracy from 0.734 to 0.767 using a single source site. That method motivates our proposal, because it shows within-class compactness helps across sites. It also exposes a problem. Pulling every diabetic retinopathy image toward one point treats a mild case and a proliferative case as the same nuisance spread that a camera change produces.

The same paper reports that biological shifts such as patient sex do not behave like color shifts. That is further evidence that not all within-class spread is noise. Porting class-level null-space projection directly would make this worse. Null Foley–Sammon projection yields at most $C-1$ useful dimensions for $C$ classes, so a binary medical task would end up with a one-dimensional embedding. Re-ID never faced this limit because it has thousands of identities.

Our claim is that a case-level nuisance null space avoids both problems. Let $f(\cdot)$ be a frozen backbone's feature, $x_i$ a training case, and $T_k$ one of $K$ acquisition perturbations. Define the nuisance scatter as

$$S_n=\sum_i\sum_k \big(f(T_k x_i)-\bar f_i\big)\big(f(T_k x_i)-\bar f_i\big)^\top,$$

where $\bar f_i$ is the mean over $k$. We seek a projection $W$ that solves

$$\max_W \ \mathrm{tr}(W^\top S_t W)\quad \text{s.t.}\quad W^\top S_n W \approx 0,\ W^\top W=I.$$

Here $S_t$ is the total scatter of the case means $\bar f_i$, so variation between patients is preserved rather than only variation between classes. Because there are as many "identities" as training cases, the dimension of the null space is not capped by the number of classes. With a 2048-dimensional feature and many perturbations, $S_n$ may be full rank. We therefore keep the subspace spanned by its smallest eigenvalues and select its dimension $m$ on source-only validation. This is an honest relaxation of the exact null space.

What is new relative to FCCL is the variance being removed: acquisition-induced spread only, estimated from paired copies, rather than all spread around a class mean. What is new relative to Generalizing a Person Retrieval Model Hetero- and Homogeneously (HHL) is also twofold. HHL uses style-transferred positive pairs as a training loss and needs camera labels plus target images. We turn such pairs into a closed-form subspace estimate that needs neither.

Research Program

The method has three parts.

The first is the nuisance generator. Its quality decides everything, since the projection can only remove the variation the generator produces. For histopathology we reuse the StarGANv2 style bank from Image-to-image translation trained on unrelated histopathology data helps for Domain Generalization. That work trains an image-to-image translation model on unrelated TCGA slides and uses the resulting styles purely as augmentation. We instead use those styles to estimate $S_n$. For fundus and dermoscopy we use physically motivated perturbations: color temperature, vignetting, blur, JPEG compression, and contrast curves typical of different camera brands. We will compare cheap generic perturbations against the realistic style bank to measure how much generator fidelity matters.

The second part is the projection itself. We compute $W$ in closed form from eigendecompositions on features from a frozen ImageNet- or source-trained ResNet-50, then refit a linear classifier on $Wf(x)$. Working on linear deep features sidesteps the test-time cost of kernel null spaces that the Re-ID paper reports (31.3 s per query versus 1.6 s). The frozen-backbone setting matters clinically. As Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors argues, validated models are often locked by regulation, and a post-hoc linear layer is far easier to audit than retraining.

The third part is an optional end-to-end variant. Here a differentiable penalty $\mathrm{tr}(W^\top S_n W)$ is computed per mini-batch, and the backbone is fine-tuned while $W$ is recomputed periodically. This tests whether the closed-form version leaves accuracy on the table.

The analysis has two strands. The first checks preserved variation. Within the referable-retinopathy class on EyePACS, we measure whether projected features still rank the five-level severity grade: rank correlation with a linear probe trained on source. If our method keeps this ordering and FCCL degrades it, the central argument holds. The second strand separates acquisition shift from biological shift. On HAM10000's sex split, a projection estimated from acquisition perturbations should help little. If it helps a lot, the shift is more acquisition-like than assumed. If it hurts, we have removed biology, which would be an important negative finding.

Baselines

The naive reference is empirical risk minimization, meaning plain cross-entropy with standard augmentation. On Camelyon17 WILDS this "Deep-All" setup reports 62.70% test accuracy, and on EyePACS it reports 0.734 target accuracy (from the StyleAugMix and FCCL papers respectively). We rerun it with our backbone, our data splits and multiple seeds, so all numbers share the same pipeline.

The strongest augmentation competitor on pathology is StyleAugMix with TCGA styles, at 92.74% on Camelyon17 WILDS. H&E color jitter reaches 91.60% and RandAugment 82.00%. Because our projection uses the same style bank, this is the fairest and hardest comparison: same generator, used either as augmentation or as a subspace estimate. Beating it requires an improvement beyond seed variance across at least five seeds. Camelyon17 is known to be noisy across runs, so we report the spread, not just means.

The strongest metric-learning competitor is FCCL, at 0.767 on EyePACS. No reported number was found on Lacuna for FCCL on HAM10000 or WSSS4LUAD. We reimplement it with identical backbone, augmentation and tuning budget, giving every method the same number of hyperparameter trials on a source-only validation split. Beating FCCL means equal or higher target accuracy together with clearly better preservation of severity ordering.

Four ablations isolate the key ingredient. The first is class-level null-space projection, which we expect to collapse to $C-1$ dimensions and lose. The second is a random projection of the same dimension $m$. The third estimates $S_n$ from generic rather than realistic perturbations. The fourth is a consistency-loss model trained on exactly the same perturbation pairs, discussed below.

Why not just…?

Why not just train with the same augmentations plus a consistency loss? This is the real threat. A loss that makes $f(T_k x)$ match $f(x)$, in the spirit of HHL's style-transferred positives, uses the same information we do. Our hypothesis is that an explicit projection has two advantages. It applies to a frozen model without retraining. It also removes nuisance directions exactly rather than trading them against the classification loss. Neither advantage is guaranteed. The matched consistency-loss ablation, using the same pairs, budget and seeds, settles the question. If it matches us on accuracy, the remaining contribution shrinks to the frozen-model and auditability setting, and we will say so.

Why not just use FCCL or supervised contrastive compactness? These methods do help across sites. However, they define the nuisance as all within-class spread, which FCCL's own observations about biological shift call into question. The severity-ordering probe on EyePACS directly tests whether that conflation costs anything.

Why not just use stain normalization or color jitter? On Camelyon17, H&E jitter already reaches 91.60%, so on pathology the headroom is small and we may only tie. This is why fundus and dermoscopy matter: there, simple color alignment is weaker, and FCCL's own baseline used standard augmentation and still dropped from 0.799 source to 0.734 target accuracy.

Why not just use domain adaptation with target images, such as DANN or CDAN? These methods need unlabeled target images, which privacy rules often forbid. FCCL already reports beating DANN and CDAN on WSSS4LUAD while using no target data, so we compare against FCCL as the stronger target-free method rather than rerunning those.

Proposed Experiments

We evaluate on three tasks: Camelyon17 WILDS (accuracy on held-out hospitals), EyePACS with a cross-camera split (target accuracy, AUC, expected calibration error and severity rank correlation), and HAM10000 across patient sex (balanced accuracy). Every model is trained on a single source domain, and model selection uses source-only validation.

Four results would be decisive. First, the case-level projection should beat class-level null space and random projection on every dataset; otherwise the Re-ID mechanism adds nothing. Second, it should match or beat FCCL's 0.767 on EyePACS while preserving severity ordering better; this is the core claim. Third, it should be within seed noise of, or above, the matched consistency-loss model; this decides whether the projection itself matters. Fourth, gains should shrink on HAM10000's sex split compared with the acquisition splits, which would confirm that the method removes acquisition variation rather than biology.

As a secondary check of generality, we will estimate $S_n$ on pooled features from the encoder of a 3D-UNet on AMOS and test on unseen MRI scanners, where the benchmark reports drops of over 20% mean Dice. This is reported as exploratory, since segmentation features are spatial and the case-as-identity mapping is less clean there.