Low-Rank Adversarial Alignment for Cross-Hospital Whole-Slide Classification

Pathology models trained at one hospital often fail at another. The tissue is the same, but stains, slide preparation and scanners differ, and a classifier picks up those differences. One classic remedy is adversarial domain adaptation. A second network, the "discriminator," tries to guess which hospital a feature came from, and the feature extractor is trained to fool it. On real medical shifts this remedy has a poor record: in the WILDS 2.0 benchmark, adversarial methods often do no better than plain supervised training. This proposal tests a specific explanation. Adversarial gradients fail because they may change every weight of the network, so they erase useful signal along with scanner signal. If those gradients are confined to a small set of low-rank "adapter" matrices in the early layers, kept separate from the adapters that learn the diagnostic task, alignment should remove scanner style while leaving diagnosis intact. Low-rank adaptation (LoRA) means adding a trainable product of two thin matrices to each frozen weight.

Whole-slide images make this hard to build. A slide holds tens of thousands of image patches. Attention that relates every patch to every other costs memory quadratic in that number, so most slide models freeze the patch encoder and cannot adapt it at all. Our design uses an asymmetry. The adversarial objective needs no slide labels and works on individual patches, so it can be trained cheaply on millions of target-hospital patches. The slide-level task is handled by a linear-cost aggregator over the full bag, and task gradients pass through only a small, attention-selected subset of patches. The two-sentence version is this: put the anti-scanner adversary into its own low-rank adapter and keep it away from the task adapter, and report whether that makes adversarial adaptation finally beat strong non-adversarial baselines on cross-hospital slides.

Core Claim

The claim has three parts. First, scanner and stain shift in histology can be absorbed by a low-rank update to the early layers of a patch Vision Transformer. Second, routing adversarial gradients only into that "style" subspace, with a separate "task" subspace in the later MLP layers, avoids the representation damage that makes full-parameter adversarial training underperform plain training. Third, a constrained adversary is far less sensitive to the adversarial loss weight $\lambda$. That weight normally has to be tuned on labelled target data, which the unsupervised setting does not have, so this is the practical bottleneck the WILDS authors identify. Concretely, each adapted weight becomes

$$W = W_0 + B_s A_s + B_t A_t,$$

where $W_0$ is the frozen pretrained weight, $B_s A_s$ is a rank-$r_s$ style adapter placed in early blocks that receives gradients only from the reversed discriminator loss and the source task loss, and $B_t A_t$ is a rank-$r_t$ task adapter placed in later MLP blocks that never receives adversarial gradients. We use the low-rank factorization and zero-initialization of LoRA unchanged. Layer placement follows the finding in eWSI that later MLP layers move most during transfer to pathology. The gradient reversal layer, which flips the sign of the discriminator's gradient before it reaches the encoder, is the standard device described in the file (Yu et al.).

Relation to Prior Work

The closest work splits into two halves that have never been joined. Double adversarial domain adaptation for WSI classification places gradient-reversal discriminators at both the patch and slide level of a Fisher-vector pipeline. It raises accuracy from 25% to 75% when transferring from Alberta to the Warwick HER2 data, but it uses a fixed CNN and has no notion of constraining where alignment acts. eWSI shows that LoRA makes end-to-end encoder tuning on whole slides feasible on one consumer GPU. However, it trains only on in-domain labels, subsamples 512 patches, and itself reports probability bias and false positives when lesions are small. Our method reuses eWSI's machinery and the double-adversarial idea of patch-level alignment. It differs from both by separating style and task subspaces and by testing robustness to $\lambda$.

The motivating negative result is WILDS 2.0. There, adversarial adaptation frequently fails to beat plain training, while self-training (Noisy Student) lifts Camelyon17 out-of-distribution accuracy from 70.8% to 86.7%. That paper is the main constraint on this proposal: any adversarial method must clear the self-training number, not just plain training. The hypothesis that low-rank updates regularize comes from Kong et al., who report that LoRA limits overfitting to spurious domain artifacts in vision attention, and Xu, who report that it preserves calibration. Neither tested adversarial objectives.

On the aggregation side, TransMIL and CAMIL use Nyström-approximated linear attention over full bags. ILRA-MIL compresses bags through a low-rank latent bottleneck. Querent computes dense attention only over query-selected regions and reports that this cuts memory from about 37GB to 1GB at 100,000 patches. All four run on frozen features. Querent notes that its accuracy is bounded by the frozen encoder, which is exactly what a scanner shift degrades. We use their aggregators as components rather than competing with them. Maxsoft is a warning that complex attention can hurt sensitivity to small lesions, so a max-style aggregator is included as an alternative head.

Research Program

The system has three components trained jointly. The first is a patch ViT, initialized from a histology self-supervised checkpoint, carrying the style and task adapters described above. The second is a small discriminator that receives patch embeddings from the output of the style-adapted blocks and predicts hospital of origin. Its inputs are drawn from all available slides at both hospitals, with no slide labels needed. Patches are weighted by the current aggregator's attention score, so that alignment concentrates on tissue the classifier actually uses rather than on background and fat, which dominate any slide. The third is a linear-cost slide aggregator, either Nyström attention as in TransMIL or Querent-style region selection, which sees every patch of a source slide.

To make slide-level backpropagation fit in memory, all patches are first encoded without gradient. Task gradients then flow back through the encoder only for the top-$K$ attended patches plus a small random set. At test time the full bag is always used, which avoids eWSI's sampling bias at inference. Cached features for the non-gradient patches are refreshed every epoch.

Three analyses accompany the method. The first sweeps adapter rank and placement to locate where scanner information lives. We measure how well a held-out linear probe can recover hospital identity from each layer before and after alignment, and whether diagnostic information, measured by a probe on patch labels where these exist, survives. The second sweeps $\lambda$ over two orders of magnitude for the constrained and unconstrained variants, and reports the spread of target performance, since lower sensitivity is a central claim. The third repeats the main comparison on a frozen pathology foundation encoder with only the aggregator and adapters trained. This tells us whether the benefit persists when the base features are already strong.

Baselines

The naive reference point is source-only training. The same encoder and aggregator are trained on source labels with standard stain and colour augmentation, with no target data used. For HER2 transfer, the double-adversarial paper reports 25% target accuracy for its source-only model. Our own source-only number is the floor on every benchmark.

The strongest non-adversarial competitor is self-training on target data. It matters most because WILDS 2.0 shows Noisy Student reaching 86.7% out-of-distribution accuracy on the Camelyon17 patch track against 70.8% for plain training. We run it with the same LoRA task adapter and the same GPU budget. Pseudo-labels are taken at slide level from the aggregator, and at patch level from attention-thresholded patches. To beat it, our method must exceed self-training on target-hospital slide AUC, averaged over held-out center splits, by more than seed variance.

The strongest existing adversarial method is the double adversarial approach. Its reported 75% target accuracy on the Alberta CCI to Warwick HER2 transfer (Double adversarial domain adaptation) is the number to beat on that split, using the same 24 unlabelled and 28 test target slides. We also reimplement its two-level gradient-reversal scheme on our ViT so that backbone strength is controlled.

The strongest frozen-feature competitor is a pathology foundation encoder with Maxsoft pooling. On Camelyon17, Maxsoft reports a WSI-level AUC of 1.000 with Prov-GigaPath. It is not clear from the file that this used a leave-one-hospital-out protocol, so we rerun it under ours. CAMIL's reported 88.1% AUC on Camelyon17 is a second frozen reference.

For in-domain sanity, eWSI's 98.5% Camelyon16 AUC is the target that our encoder-tuning pipeline should roughly reproduce before any adaptation is added.

Two ablations isolate the key ingredient. The first is unconstrained adversarial training, where the reversed gradient reaches all adapters or the full network. The second is a single shared adapter that receives both the adversarial and task losses. If the separated style subspace does not beat both under identical rank budget, tuning budget, and $\lambda$-selection protocol, the core claim fails.

All methods get the same number of tuning trials. $\lambda$ and other hyperparameters are chosen without target labels, using source validation loss or the discriminator-accuracy heuristic. Oracle target-label tuning is reported separately as an upper bound.

Why not just…?

Why not just use plain training with strong augmentation? WILDS 2.0 shows this is often as good as adaptation methods, and Maxsoft's per-patch augmentation is a cheap, reproducible version of it. On Camelyon17, however, the same benchmark records a 15.9-point gap that self-training closes and augmentation does not. The HER2 transfer collapses to 25% without adaptation. We include augmentation in every arm, so any gain we report is on top of it.

Why not just use a frozen pathology foundation model with simple pooling? This is a real threat. A 1.000 AUC on Camelyon17 suggests that benchmark may be close to saturated for very large encoders. We respond in two ways. We test under strict leave-one-hospital-out splits and on the much harder HER2 immunohistochemistry transfer, where the stain itself differs. We also run the frozen-foundation analysis described above. If adaptation adds nothing on top of a strong frozen encoder, we report the method as useful mainly for smaller encoders and compute-limited settings. Maxsoft notes that a small domain encoder already uses about 1,000 times less computation, which keeps that a meaningful regime.

Why not just self-train instead of training an adversary? Self-training is the method to beat. Its weakness here is that slide-level pseudo-labels are few: there are only 24 unlabelled target slides in the Warwick split, while patch labels do not exist. The adversary, by contrast, uses millions of unlabelled patches. We test self-training head-to-head and in combination with our method. If self-training alone matches the combination, the adversarial component is unnecessary, and we will say so.

Why not just subsample 512 patches as eWSI does? eWSI itself reports that sampling biases probabilities toward false positives when lesions are small. We compare full-bag and sampled inference on Camelyon16, where lesions are tiny, and report calibration error alongside AUC.

Proposed Experiments and Decisive Checks

Benchmarks are Camelyon17, run both at slide level with each of the five hospitals held out in turn and on the WILDS patch track; the Alberta CCI to Warwick HER2 transfer; and Camelyon16, used for in-domain sanity and for localization Dice against CAMIL's reported 0.515. Metrics are target slide AUC and accuracy, expected calibration error, localization Dice, and peak GPU memory and wall-clock time on a single 24GB GPU.

The decisive checks are three. The first is whether the separated style adapter beats both unconstrained adversarial training and self-training on target AUC without target-label tuning. The second is whether its target performance varies over the $\lambda$ sweep with a markedly smaller spread than the unconstrained adversary. The third is whether hospital identity becomes unrecoverable from aligned features while patch-level tumour information is preserved. A failure of the first check with success on the second would still be a useful finding: low-rank constraint would then fix adversarial training's instability but not its ceiling.