Large pretrained vision models are usually adapted to new domains by freezing their weights and training a small add-on, such as a prompt or an adapter. This fails when the frozen model has no internal features for what the new domain looks like: no amount of re-weighting can recover structure the encoder never learned to see. Prior work treats this failure, called representation rigidity, as a property of the whole image. Driving scenes and medical scans suggest otherwise. At night, the road surface and sky may be well represented while glare-smeared pedestrians are not. In CT, soft-tissue texture may transfer while bone-adjacent organ boundaries do not. If rigidity is local, the add-on capacity, and the training signal that drives it, should go only to the patches where the frozen model is blind.
We propose to locate those patches with a tool the backbone already has. A Vision Transformer (ViT, a model that cuts an image into a grid of patches and lets every patch attend to every other) that was pretrained by masked image modeling (hiding most patches and learning to reconstruct them) comes with a reconstruction head. On a target image, a patch the frozen model cannot reconstruct from context, measured relative to how well it reconstructs comparable source patches, is a patch its features do not cover. We use this per-patch "rigidity map" to send only blind patches through a trainable side network that is cut off from the backbone's gradients. Easy patches pass through the untouched frozen path. The bet is that this map predicts where adaptation actually helps better than the prediction-uncertainty maps the field uses now, and that routing by it matches broad adapters at a fraction of the cost.
Core Claim
The claim is that representation rigidity is spatially heterogeneous and can be measured in a single cheap pass. The measure is the frozen encoder's own masked-reconstruction failure, normalized against the source domain. It is distinct from prediction uncertainty: uncertainty also fires on patches that are well represented but semantically ambiguous, such as the road-versus-sidewalk boundaries that MIC resolves with context alone.
Concretely, let $e_i$ be the reconstruction error of patch $i$ when masked, computed with the frozen encoder and its pretraining decoder on structural (Histogram of Oriented Gradients, HOG) targets. HOG targets are used because ADMA shows they suppress illumination and weather artifacts that raw pixels would penalize. Let $\mu_c, \sigma_c$ be the mean and spread of that error over source patches whose frozen prediction is class $c$. The rigidity score is
$$r_i = \frac{e_i - \mu_{c(i)}}{\sigma_{c(i)}},$$
where $c(i)$ is the frozen model's predicted class for patch $i$. Normalizing per class matters because some content, such as foliage, is intrinsically hard to reconstruct in every domain. That intrinsic difficulty is not a domain gap.
The adapted feature for patch $i$ is the frozen feature plus a gated side-network output, $z_i = f_i + \mathbb{1}[r_i > \tau]\, g(f_i, \mathrm{sg}(F))$. Here $f_i$ is the frozen final-layer token, $F$ is the set of frozen intermediate tokens with stop-gradient $\mathrm{sg}$, $g$ is the trainable side network, and $\tau$ is a threshold set by a token budget (for example, the top 15% of patches).
What is new relative to the closest work is the source of the routing signal and what it controls. ADMA selects patches with Monte Carlo (MC) Dropout uncertainty and then updates the whole model. SVDP uses the same kind of uncertainty to place sparse prompts in pixel space on a frozen model. Neither measures whether the encoder can represent a patch, and neither puts new capacity only where representation is missing.
Research Program
The prior work plays three roles in this proposal.
Work that motivates it. Mind The Gap shows on cardiac MRI-to-CT that global style transfer stalls near 58% Dice, and that local alignment of hard structures pushes it above 80%. Fei Yu et al. and the domainness maps of Tommasi et al. establish that domain shift can be scored spatially. On the rigidity side, Learning Federated Visual Prompt in Null Space, Bridging the Pathology Domain Gap and Few-shot Adaptation of Medical Vision-Language Models show that prompts and adapters fail when target features lie outside the frozen span. All of these treat rigidity as global.
Work that supplies the method. Res-Tuning supplies the detached bypass, which avoids backpropagating through the frozen backbone and cuts training memory by up to 70% on clean benchmarks. It never routes tokens selectively and was not tested under domain shift. MIC supplies the target-side training signal: a teacher sees the full image and a student sees a masked one and must match the teacher's pseudo-labels. ADMA supplies the HOG target.
Work that constrains it. MIC reports that masked objectives degrade when the surrounding context is uninformative. For our rigidity map, this means a blind patch inside a blind neighborhood may be scored poorly. We therefore compute $e_i$ with several complementary masks and report how reliable the map is as a function of local context quality. SVDP's "spatial blackout" result rules out putting new capacity at the input as dense blocks. This is one reason the side network acts on tokens instead.
What we build. A source-supervised segmentation decoder sits on a frozen masked-image-model-pretrained ViT. On unlabeled target data, the rigidity map is computed with four complementary 75% masks. Because the encoder sees only visible tokens, each pass processes a quarter of the patches, and we will report the cost against MC Dropout's repeated full passes rather than assume it is lower. The side network $g$ is a small transformer. Its self-attention runs among routed tokens, and it cross-attends to the frozen tokens for context. Because only routed tokens enter $g$, its attention cost scales with the routed budget, not the full patch count.
How it is trained. Training uses MIC-style masked consistency on the target. In a secondary variant, the student's mask is sampled with probability increasing in $r_i$.
What we analyze. The central analysis asks whether the map finds the right patches. We train a fully fine-tuned MIC model as an oracle for where adaptation helps. Each patch's "adaptation headroom" is the change in pixel correctness between the frozen model and that oracle. We then ask how well $r_i$, the frozen head's predictive entropy, and MC Dropout variance each rank patches by headroom.
Baselines
Do nothing. The frozen backbone with its source-trained decoder and no target adaptation fixes the floor. On MMWHS, prior work reports the unadapted figure as 17.12% Dice (Mind The Gap) and 13.2% (PnP-AdaNet). We will rerun it on our own backbone.
Strongest existing method: MIC with full fine-tuning. MIC reports 75.9 mIoU on GTA→Cityscapes and a +4.3 mIoU gain on DarkZurich. We run its released code (lhoyer/MIC) on our backbone with the same unlabeled target data, iterations and augmentation. It is the strongest competitor because it removes rigidity outright by updating every weight. We therefore treat it as a near-ceiling, not a target to beat. Success means coming within roughly one mIoU of it while training under 5% of the parameters, with substantially lower training memory.
Strongest frozen-backbone competitors: SVDP and ADMA. SVDP reports 60.1% mean mIoU on Cityscapes→ACDC. ADMA reports 61.8% mIoU on Cityscapes→ACDC. Both numbers are under a continual test-time protocol, so we rerun both in our offline setting and also run our method in their continual protocol for a direct comparison. ADMA must be adapted to update only an adapter of matched size, and we report both the matched and original versions.
Medical reference. Mind The Gap's 82.42% Dice on MMWHS MRI→CT is the reference there. It uses a convolutional network with style transfer, so the comparison is indicative, not controlled.
Isolating the key ingredient. No reported result was found on Lacuna for Res-Tuning-Bypass under domain adaptation. The decisive ablation is therefore the same side network applied to all patches, trained with the identical MIC loss and tuning budget. A second ablation keeps the routing but swaps the routing score for entropy and for MC Dropout variance. Beating these ablations is what would show that the rigidity map, not the bypass itself, carries the result.
Why not just…?
Why not just put the bypass or a LoRA on every patch? This is the most serious threat. A side network applied everywhere has strictly more capacity, and if it matches routed accuracy, the routing contributes only efficiency. We will not argue around this.
The proposal claims two things. The routed version should match the all-patch version's accuracy at much lower side-network cost, because attention among routed tokens scales with budget squared. It should also beat the all-patch version on easy regions, where updates driven by pseudo-labels risk the background negative transfer documented by Chaoqi Chen et al.. The per-region breakdown, split into low-$r$ and high-$r$ patches, settles which claim survives. If neither does, the contribution reduces to the diagnostic map.
Why not just route by prediction entropy or MC Dropout, as SVDP and ADMA do? Entropy is free and is the obvious single-pass alternative. Our argument is that uncertainty conflates ambiguity with blindness: the ambiguous road-sidewalk boundaries in MIC are well represented and fixable by context, while SVDP itself notes MC Dropout's repeated-pass overhead. The headroom-ranking analysis tests this directly. If entropy ranks headroom as well as $r_i$, we adopt entropy and drop the reconstruction machinery.
Why not just fine-tune everything, as MIC does? It works, and it is our ceiling. However, full updates on small or shifting targets cause forgetting, a problem reported in pathology by Bridging the Pathology Domain Gap and in continual ACDC streams by SVDP and ADMA. Full updates also need one dense checkpoint per domain. We measure source-domain retention and per-domain storage alongside accuracy.
Why not just use global alignment or image translation? The evidence is already against it for these shifts. On MMWHS, DANN reaches 34.5% and ADDA 37.2% Dice (PnP-AdaNet), and CycleGAN reaches 56.61% (Mind The Gap). We include CycleGAN only as a reference.
Proposed Experiments
Driving benchmarks. We use Cityscapes as source (Cityscapes), with the four ACDC adverse conditions as targets (ACDC) and DarkZurich as a nighttime target. We report mIoU overall and per condition.
Medical benchmarks. We use MMWHS MRI→CT and CT→MRI with per-structure Dice for ascending aorta, left atrium, left ventricle and myocardium. We add CHAOS abdominal MRI→CT, using a TransUNet-style decoder (TransUNet) on the frozen ViT.
Costs. We report trainable parameters, peak training memory, and wall-clock time to compute the map versus MC Dropout.
First decisive check. Per-patch headroom ranking: the area under the curve for $r_i$ versus entropy versus MC Dropout, stratified by condition. We expect the largest margin at night and in fog, where frozen features plausibly go blind, and near zero in rain, where shift is mostly appearance noise. A uniform margin, or none, would contradict the premise that rigidity is local.
Second decisive check. We sweep the token budget from 5% to 100%. If the method is right, accuracy should saturate well before 100% and the routed curve should sit at or above the all-patch bypass. If accuracy rises steadily to 100%, rigidity is global and the routing idea fails.
Third decisive check. Ablating the per-class source normalization tests whether raw reconstruction error merely tracks texture. ACDC's invalid-pixel masks let us check whether high-$r$ patches are disproportionately the genuinely ambiguous pixels. If they are, the map is partly measuring irreducible ambiguity and should not drive capacity there.
References
- MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation
- Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation
- Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction
- Mind the Gap: Alleviating Local Imbalance for Unsupervised Cross-Modality Medical Image Segmentation
- DAST: Unsupervised Domain Adaptation in Semantic Segmentation Based on Discriminator Attention and Self-Training
- Learning the Roots of Visual Domain Shift
- Learning Federated Visual Prompt in Null Space for MRI Reconstruction
- Bridging the Pathology Domain Gap: Efficiently Adapting CLIP for Pathology Image Analysis with Limited Labeled Data
- Few-Shot Adaptation of Medical Vision-Language Models
- Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone
- PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network with a Benchmark at Cross-modality Cardiac Segmentation
- Harmonizing Transferability and Discriminability for Adapting Object Detectors
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding