A segmentation model trained on CT scans usually fails on MRI of the same organs, because the two machines measure different physics. Hospitals cannot wait for retraining, and many uses, such as live ultrasound or intraoperative video, give the model only one frame and roughly 30 milliseconds to answer. The standard fast fix, test-time adaptation (nudging a few model parameters on each incoming image without labels), was built for convolutional networks. It updates batch-normalization statistics, which are noisy when estimated from a single image. Transformer backbones, which now dominate medical foundation models, have no comparable cheap knob.
We propose that query-key normalization supplies that knob. Query-key normalization (QK-norm) rescales each attention head's query and key vectors to fixed length before comparing them. Once a model is trained this way, an attention score is a cosine similarity times a learned per-head scale, so the scale behaves like a temperature: it sets how sharply each head concentrates its attention. Our claim is that much of the damage from cross-modality shift in such a network shows up as per-head drift in attention sharpness, which can be measured as attention entropy. That drift can be undone without any backpropagation. For each head, a single scalar temperature is solved so that the head's entropy on the incoming frame matches its average entropy on source data. The update costs a few scalar root-finding steps per head, which fits inside a video frame budget that gradient-based test-time adaptation does not.
Core Claim
Take a head $h$ with normalized query and key vectors $\hat q_i, \hat k_j$ (unit length). Its attention weights are $a_{ij} = \mathrm{softmax}_j(\tau_h \, \hat q_i^\top \hat k_j)$, where $\tau_h > 0$ is the head's temperature. For fixed cosine scores, the mean attention entropy $H_h(\tau_h)$ falls monotonically as $\tau_h$ grows. Given a reference entropy $\bar H_h$ recorded on source-modality training data, the test-time update is
$$\tau_h^{\text{test}} = \text{the } \tau > 0 \text{ such that } H_h(\tau; x_{\text{test}}) = \bar H_h,$$
where $x_{\text{test}}$ is the incoming frame. Monotonicity makes this a one-dimensional bracketed root problem that a handful of bisection or Newton steps solve. On a video stream, the solution is smoothed across frames with an exponential moving average.
The claim depends on QK-norm. In an unnormalized transformer, a change in modality changes token magnitudes, and magnitude and direction are entangled in the logits. A temperature knob then fights a moving, query-specific scale rather than one shared degree of freedom per head. The proposal therefore makes two falsifiable predictions. First, entropy matching recovers a large share of the cross-modality Dice loss in QK-normed networks. Second, the same procedure does much less in a matched network trained without QK-norm.
Relation to Prior Work
Three groups of papers motivate the idea. Scaling Vision Transformers to 22 Billion Parameters and Unified-IO 2 introduced QK-norm to stop attention logits from blowing up during large-scale and heterogeneous multimodal training. Both treat it purely as a training stabilizer. Neither uses the resulting temperature as something to adjust at deployment, and neither touches medical imaging or latency.
Text Capability Loss in Vision-Language Adaptation found three things:
- Cross-modal fine-tuning moves query and key weights 3–4x more than unimodal tuning.
- A per-head attention statistic, sink strength, predicts downstream damage with Spearman correlation 0.97.
- Per-head QK-RMSNorm protects the model where layerwise normalization does not.
This is the closest evidence that cross-modal shift lives in per-head attention geometry. We borrow its per-head design. The same paper also constrains us: inserting QK-norm after pretraining fails. Our backbones are therefore trained with QK-norm from the start, and the no-QK-norm control is a separately trained twin, not an ablated copy.
One QK Channel, Many Sources proposed QK-Guard, a controller that triggers normalization during training when logits pass a hand-picked threshold of 30. We act at inference rather than during training. We also drive each head toward a data-derived reference entropy instead of a fixed threshold, which is one answer to the open problem of choosing such thresholds a priori.
Dissecting Query-Key Interaction in Vision Transformers showed that early heads group same-object tokens while late heads attend across context. This gives a testable sub-hypothesis. Intensity and texture shifts between CT and MRI should mostly disturb the early grouping heads, so recalibrating early layers alone may capture most of the gain.
The method is the transformer counterpart of convolutional test-time adaptation. TASD adapts batch-norm affine parameters by backpropagation against a shape dictionary in 56 ms per slice, and only within one modality family. InTEnt avoids noisy single-image gradients by ensembling interpolated batch-norm statistics, using entropy computed separately over foreground and background, in about 60 ms. You Point, I Learn adapts on clinician clicks in 50–90 ms. We share InTEnt's view that single-image gradients are fragile, so we avoid gradients altogether. Unlike all three, we target transformers, physics-different modality pairs, and a sub-frame budget.
Adapting Vision Foundation Models for Real-time Ultrasound Image Segmentation shows that transformer segmenters can reach about 77 FPS with TensorRT. It adapts across ultrasound tasks, not across modalities, and has no deployment-time adaptation.
Research Program
Backbones. We build a compact hierarchical vision-transformer segmenter sized to run under 33 ms per 2D frame, in two twins trained with identical data, schedule, and compute: one with per-head QK-RMSNorm and a learnable per-head temperature, one with standard scaled dot-product attention. During source training we log each head's mean attention entropy on held-out source data; these are the references $\bar H_h$.
Cost of measuring entropy. Fused attention kernels never build the full attention matrix. We therefore estimate entropy on a random subsample of queries per head, using the identity that entropy equals the log-sum-exp of the logits minus the softmax-weighted mean logit. We will report the overhead directly rather than assume it is small.
Diagnostic study, before any adaptation results are claimed. On CT→MRI transfer we measure three things:
- whether per-head entropy drift is larger in the unnormalized twin;
- whether drift, especially in early layers, correlates with per-volume Dice drop, in the spirit of the sink-strength correlation of Choi et al.;
- how much Dice an oracle recovers when it tunes each $\tau_h$ using target labels.
The oracle is the ceiling. If it recovers little, temperature is the wrong knob and the project stops there.
Baselines
The naive reference is the QK-normed twin applied to the target modality with no adaptation. It shares every weight with our method, so any gain is attributable to the temperature update alone. A second naive reference is the unnormalized twin, also without adaptation, which tells us whether QK-norm by itself already helps cross-modality transfer before any adaptation.
The strongest gradient-based competitor for transformers is single-step entropy minimization on LayerNorm affine parameters. This is the direct analogue of batch-norm test-time adaptation. We run it on the same QK-normed backbone and give it the same latency budget. We tune its learning rate and step count on the same small held-out target-free source validation split we use for our smoothing constant, with a matched number of tuning trials.
The strongest published convolutional methods are TASD and InTEnt. They are evaluated on their own benchmarks using their reported protocols:
- TASD reports 81.1% mean Dice on prostate MRI across unseen sites, against 78.2% for M-ADA and 74.8% unadapted (TASD).
- InTEnt reports 71.6% average Dice across spinal cord MRI, chest X-ray, and fundus splits, against 68.7% for SITA (InTEnt).
We also re-implement InTEnt-style entropy-weighted ensembling over interpolated LayerNorm statistics on our backbone, so the comparison holds the backbone fixed.
For CT→MRI on CHAOS without target labels, no reported result found on Lacuna. The only CHAOS numbers in hand are supervised: 0.98 Dice single-modality CT, 0.95 MRI, and 0.88 when one model is trained jointly on both (CHAOS). We treat the joint-training figure as a label-using reference point, not a bar a zero-label method is expected to clear.
The key ablation isolates QK-norm. We apply the identical entropy-matching temperature update to the unnormalized twin. We will count the proposal as successful if three conditions hold:
- On CT↔MRI it beats LayerNorm-affine entropy minimization in Dice at lower latency.
- It stays within a small margin of TASD and InTEnt on their home benchmarks while running faster.
- The ablation shows a clearly smaller gain without QK-norm.
Why not just…?
Why not just adapt LayerNorm parameters with entropy minimization? This is the most natural transformer analogue of existing methods and a real threat. InTEnt's motivation is that single-image gradient updates of normalization parameters are noisy (InTEnt). Choi et al. also show that cross-modal shift concentrates in query-key geometry, which LayerNorm affine parameters only touch indirectly. Neither finding proves that LayerNorm adaptation loses. The head-to-head comparison at equal latency settles it. If LayerNorm adaptation matches us at equal cost, our contribution shrinks to a cheaper, gradient-free variant, and we would report it that way.
Why not just train with heavy intensity augmentation and skip test-time adaptation? Randomizing contrast, gamma, and noise can make a model insensitive to the intensity differences between CT and MRI. It is also cheaper than any adaptation. We apply the same augmentation to every arm, including ours. The question then becomes whether temperature recalibration adds anything on top of an augmented model. If augmentation erases the entropy drift measured in the diagnostic study, the method has no role, and the diagnostic will show that early.
Why not just fix one temperature per modality from a few target images? If the drift is a stable property of the modality rather than of the individual frame, a one-time offline calibration would be simpler and free at runtime. We include this baseline: per-head temperatures fit on five unlabeled target volumes, then frozen. Per-frame updating must beat it on held-out target data, especially across MRI sequences (T1-DUAL versus T2-SPIR) and patients, to justify running online.
Why not just translate images with CycleGAN first? Cross-Modality Knowledge Transfer for Prostate Segmentation from CT Scans raised real-CT prostate Dice to 0.73 using an MRI-to-synthetic-CT pipeline. A residual gap remained against the 0.80 obtained on its synthetic test set. Running a translation network on every frame adds a full generator pass, and the paper reports no latency. We report its Dice as an offline reference but do not treat it as a real-time competitor.
Proposed Experiments and Decisive Checks
Primary cross-modality setting. We use CHAOS: train on CT, test on T1-DUAL and T2-SPIR MRI for liver, kidneys, and spleen, and also run the reverse direction (CHAOS). A second source is ABD-30 CT, tested on the CHAOS MRI organs. We report Dice and maximum symmetric surface distance, since CHAOS shows boundary errors around 20–30 mm persist even when Dice looks high.
Within-modality transfer. We test on prostate MRI across sites against TASD's 81.1%, and on InTEnt's spinal cord, chest X-ray, and fundus splits against its 71.6%.
Streaming setting. We test across sites on CardiacUDA and CAMUS ultrasound sequences, where the moving-average temperature is exercised frame to frame.
Latency. We measure median and 99th-percentile milliseconds per frame on an A5000-class GPU, matching the hardware used by OAIMS, and on one embedded GPU. The update must fit within a 33 ms total budget, compared with 56–90 ms for the gradient-based methods cited above.
Decisive checks, in order of how fatal they are.
- The oracle-temperature ceiling must recover a meaningful share of the CT→MRI Dice drop. If it does not, the knob is wrong.
- The entropy-matched temperature must close most of the gap between no adaptation and the oracle. If it does not, entropy is the wrong target.
- The unnormalized-twin ablation must gain clearly less. If it does not, QK-norm is incidental and the claim reduces to generic temperature scaling.
- Restricting recalibration to early layers should keep most of the gain, as the grouping-versus-contextualization picture of Pan et al. predicts. A failure here would not kill the method, but it would tell us that shift is not confined to the grouping heads.
- On ultrasound streams we check that the smoothed temperatures do not drift without bound over long sequences. Per-frame failure rate under probe motion and occlusion is reported alongside Dice.
References
- Scaling Vision Transformers to 22 Billion Parameters
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
- Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
- One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
- Dissecting Query-Key Interaction in Vision Transformers
- Single-Domain Generalization in Medical Image Segmentation via Test-Time Adaptation from Shape Dictionary
- Medical Image Segmentation with InTEnt: Integrated Entropy Weighting for Single Image Test-Time Adaptation
- You Point, I Learn: Online Adaptation of Interactive Segmentation Models for Handling Distribution Shifts in Medical Imaging
- Adapting Vision Foundation Models for Real-Time Ultrasound Image Segmentation
- CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation
- Cross-Modality Knowledge Transfer for Prostate Segmentation from CT Scans