Does Pre-Norm Carry the Scanner Into the Prediction?

Medical imaging models often fail when moved to a scanner from a different manufacturer. Each device leaves its own fingerprint in the image: its gain, speckle texture, contrast curve and filtering. Networks pick up these fingerprints as shortcuts. The field usually treats the network as a fixed feature extractor and fixes the problem outside it, with adversarial losses, image harmonization, or post-hoc mapping of embeddings. We ask a question about the architecture itself. Most Vision Transformers use pre-layer normalization (Pre-LN): each block normalizes only its own input, and its output is added to an untouched running sum called the residual stream. Under this design, whatever the patch-embedding layer writes into that stream reaches the final layer by an uncorrected direct path. That includes low-level intensity and texture statistics, which are exactly where scanner fingerprints live.

Our claim is that this direct path is a measurable, localized carrier of device identity, and that a cheap change can largely cut it. The change is to re-normalize the residual stream once, early in the network, rather than redesign normalization everywhere. In neural machine translation, Exploring the Impact of Layer Normalization for Zero-shot Neural Machine Translation showed something similar: post-layer normalization (Post-LN, which normalizes the stream after every residual addition) strips source-language identity from intermediate layers and beats Pre-LN by up to 12.3 BLEU on unseen language pairs. Nobody has tested whether the same mechanism governs hardware shift in continuous images. Nobody has tested where in depth the reset has to happen either, and that question is what makes the result usable.

Core Claim

We make three predictions about Vision Transformers trained on multi-device medical images. First, vendor identity is linearly decodable from the Pre-LN residual stream at every depth. In Pre-LN the final stream is exactly the sum of the embedding output and each block's output, so we can decompose it term by term. We predict that a large share of the decodable vendor signal sits in the embedding term rather than in the block outputs. Second, inserting a single Post-LN-style reset of the stream after block $k$ lowers vendor decodability and improves zero-shot cross-device accuracy most when $k$ is small. Resets placed deep should help little, because by then the shortcut has already been read and amplified. Third, the reset helps more as the network gets deeper. This follows because the share of the stream contributed by deep blocks shrinks as the stream's norm grows with depth, the "lazy deep layer" effect described in Mix-LN.

If all three hold, the practical recipe is "post-norm early, pre-norm late". That is the Mix-LN layout, but the motivation is invariance rather than gradient flow. If none hold, the result is still useful: it would show that the NMT finding does not transfer to sensor shift, and that architecture-external fixes remain necessary.

How This Sits Against Prior Work

The motivating evidence comes from two places. Mao et al. found the shortcut mechanism in translation, but only for discrete tokens and only as a global Pre-LN versus Post-LN swap. They did not localize the effect by depth and did not study images. The second source is the feature-space translation study on AI-READI. It found that embeddings from the Pre-LN retinal foundation model RETFound retain device shifts between Topcon Triton and Maestro2. It then aligns them post hoc with tabular models and deliberately leaves the backbone frozen. Our proposal tests the backbone choice that study set aside.

Three papers supply methods we reuse. Mix-LN puts Post-LN in the first fraction of layers and Pre-LN in the rest. SiameseNorm runs a post-norm stream and a pre-norm stream through shared blocks, and it already reports a 1.5% top-1 gain for DeiT-S on ImageNet. SpanNorm adds a spanning residual connection with a block-level variance reset. All three were evaluated for perplexity, downstream accuracy, and stability, never for shift across acquisition devices.

Two lines of work constrain the design. Post-LN diverged at 1B parameters in the Mix-LN experiments, so pure Post-LN at depth is a weak arm and needs warmup to be fair. Anatomy of Massive Activations and Attention Sinks shows that Pre-LN streams grow extreme outlier channels that act as global references. These are a plausible competing home for vendor information, so we probe them separately.

Prior normalization-for-shift work changes how statistics are computed, not where normalization sits. Examples are TTN, Split Batch Normalization, and frequency-wise instance normalization for multi-device acoustics. These methods are mostly built on convolutional networks with batch statistics, and several need target-domain batches at test time. Our intervention needs neither.

Research Program

We pretrain matched masked-autoencoder Vision Transformers on public retinal photographs. These are the EyePACS and APTOS-style public sets used in the RETFound suite, plus the training split of AI-READI. We build ViT-S-width models at two depths, 12 and 24 blocks. Each depth gets several normalization layouts: plain Pre-LN, Post-LN with warmup, Mix-LN, SiameseNorm, and a family we call "reset-at-$k$". A reset-at-$k$ model is plain Pre-LN with one Post-LN-style normalization of the stream inserted after block $k$, for $k$ in $\{1,2,4,8\}$ at depth 12 and the matching fractions at depth 24.

All variants share the data, token budget, optimizer, and tuning budget. Each variant gets the same small learning-rate and warmup sweep, and we select on in-distribution validation loss only. Target-device data never enters model selection.

The diagnostic is a layer-wise vendor probe, the image analog of Mao et al.'s layer-wise language recognition. At each block we train a linear classifier to predict the device from mean-pooled tokens, on held-out subjects. For Pre-LN we also decompose the final stream into the embedding term and the block terms and probe each one. For outlier channels, we compare vendor decodability with the massive-activation channels masked and unmasked.

The outcome tasks come from two places. On AI-READI, which has paired same-participant images from two cameras, we regress the AutoMorph vascular biomarkers on one device and evaluate zero-shot on the other, in both directions. We report per-biomarker $R^2$ and single out tortuosity, which Rastogi et al. found resists alignment. The paired design also lets us measure invariance directly, as the cosine similarity between the two devices' embeddings for the same eye.

The second outcome task tests whether the effect is specific to fundus imaging. On FUSEP, we train detection on two of the SAMSUNG, Sonoscape and GE ultrasound systems and test on the held-out one, using a DETR-style detector on each backbone variant. Its 14 tiny structures, each occupying 0.2–0.5% of the image, stress whether a reset erases fine anatomy along with texture.

Baselines

The do-nothing reference is plain Pre-LN at each depth, trained and fine-tuned with the identical pipeline. Every gain is measured against it.

The strongest architectural competitor is SiameseNorm, because it already improves ViTs and is stable at high learning rates. If a single early reset matches it on cross-device scores, the simpler intervention wins. If SiameseNorm clearly exceeds it, invariance needs more than one reset. Mix-LN is run at its default fraction of 0.25 and at our best $k$.

The strongest method external to the architecture is a vendor-adversarial head with gradient reversal on a Pre-LN backbone, in the style of Degel et al.. It gets the same fine-tuning data and an equal tuning budget for the adversarial weight. On their 3D echo cohort, Degel et al. report that their combined adversarial and shape-prior model improves cross-vendor mean surface distance from 3.56 mm to 2.01 mm. That number is for a convolutional network on a different task and serves only as evidence that the adversarial approach is strong.

On AI-READI, the reference number is Rastogi et al.'s end-to-end cross-device biomarker estimate of $R^2$ = 76.0%, obtained from frozen RETFound embeddings plus a tabular translator trained on paired data. Our zero-shot protocol uses no paired supervision, so matching 76.0% without pairs would count as a clear win. Falling short while beating Pre-LN in the same protocol would still be a positive result. We will also run their translator on top of each of our backbones to see whether the gains add up.

For FUSEP cross-device detection, there is no reported UDA or source-free result on Lacuna. The in-distribution reference is Relation-DETR's 85.6% mAP on the CRL view. We will report the held-out-vendor drop relative to that level.

The ablation that isolates our ingredient is the reset-at-$k$ sweep, compared against a reset placed only in the final quarter of the network. Beating Pre-LN means a statistically reliable gain in held-out-device $R^2$ or mAP across three seeds, with in-distribution accuracy within one standard deviation of Pre-LN. Gains bought by lower source accuracy do not count.

Why not just…?

Why not just standardize the input images? This is the most serious threat. A per-token reset might do nothing more than remove global gain and contrast, and per-image standardization or histogram matching does that too. We run Pre-LN with per-image standardization and with randomized gain, contrast and speckle augmentation as explicit arms. If either closes the gap to the reset models, we will report that the architectural story reduces to intensity normalization. The embedding-term probe will show whether vendor information survives input standardization, as Cole et al. suggest it can even in raw MRI data.

Why not just add an adversarial vendor loss? It is a strong baseline and is included at equal budget. Two things count against it. Degel et al. needed an extra shape autoencoder because adversarial blinding also removed anatomy. It also needs vendor labels for every training image. FUSEP's tiny structures and the tortuosity biomarker are where we expect over-blinding to show up. If adversarial training wins outright there, the reset is not needed. If the two combine additively, the reset is a free complement.

Why not just harmonize or align after the fact? Image-to-image harmonization can move anatomy. The anatomy-guided Schrödinger bridge work exists precisely because diffusion bridges left 0.89-pixel boundary errors, which it reduces to 0.43. Post-hoc embedding translation needs paired multi-device data and cannot recover what the encoder discarded. Our experiment that stacks the translator on each backbone tests this second point directly.

Decisive Checks

Three results would settle the core claim. The first is whether the vendor-probe curve and the cross-device score both improve most for small $k$ and little for late resets. The second is whether the Pre-LN-to-reset gap is larger at 24 blocks than at 12. The third is whether these effects survive input standardization and augmentation.

The claim is refuted if late resets work as well as early ones, if input standardization erases the effect, or if gains appear only alongside lower in-distribution accuracy. A positive outcome would give an inexpensive design rule for future medical foundation models, together with a probe that any group can run on an existing backbone to see where it stores the scanner.