Supplying Missing Frequency Bands to Frozen Vision Backbones

Large pretrained vision models such as DINOv2 are often adapted to new settings like night-time driving or satellite imagery with their weights frozen, training only a small add-on module. This hits a ceiling that the literature calls representation rigidity: the frozen model cannot produce features it never learned, and a small module that only re-weights those features cannot recover information the backbone has already thrown away. Recent frequency-based adapters split the backbone's features into low and high spatial frequencies and tune each band separately. That helps when the right information is present but tangled. It cannot help when the backbone discarded a band of the image altogether.

We propose to measure, per frequency band, how much of the input image a frozen backbone actually retains. We then give the backbone a small side network that reads only the poorly retained bands directly from the raw pixels. The side network is cheap: it applies a learned, image-dependent frequency gate to the input image, then runs a shallow convolutional pyramid. The frozen transformer keeps supplying global semantics, and the convolutional stream supplies local detail in exactly the bands the transformer loses. The side network is trained without backpropagating through the backbone. In two sentences: frozen backbones lose specific, measurable frequency bands of their input, and that is where adapters fail. A small convolutional branch that reads those bands from the raw image, behind a learned noise-suppressing frequency gate, should fix the failure where feature-filtering adapters cannot, at lower training memory.

Core Claim

The claim has two parts, and the first is falsifiable independently of the method. First, rigidity is band-specific and predictable. A linear readout test on frozen patch features can tell, before any adaptation, which input frequency bands a backbone discards on a target domain. Adapters that operate only on backbone features should plateau precisely on tasks whose relevant bands score poorly on this test. Second, the right fix has three parts. The missing information is re-supplied from the input image. A local convolutional operator processes it, since convolutions handle the localized edge and texture content that lives in high bands. An input-conditioned frequency gate decides which bands enter, because those same high bands carry sensor noise and lighting artifacts.

Relation to Prior Work

Three adapters motivate this work by showing that frequency decomposition helps frozen backbones. They share one limitation: they act on what the backbone already outputs. NightAdapter decomposes frozen DINOv2 features with a discrete sine transform, adapts illumination-insensitive bands and randomizes sensitive ones. It reports a 13.4% mIoU improvement over the prior leading method on Cityscapes→ACDC-Night with about 11.5M trainable parameters. Earth-Adapter splits frozen-backbone features with a 2D discrete Fourier transform into low- and high-frequency adapters, combines them through a learned router, and reports a 9.0% average mIoU gain in remote-sensing domain adaptation. Visual Fourier Prompt Tuning Fourier-transforms prompt tokens but keeps only the real part, discarding phase. It reaches 73.20% mean accuracy on VTAB-1k against 69.43% for plain visual prompts. All three adapters filter backbone features or tokens. Our module filters the raw image instead. It keeps phase intact, and it uses convolutions rather than MLPs inside the frequency branch.

Two papers supply the mechanism we reuse. ViT-Adapter adds a convolutional stem that reads raw pixels and injects multi-scale spatial priors into a plain ViT. Its own Fourier analysis shows that this restores high-frequency detail the ViT loses. That is direct evidence for our premise, but its stem is full-band and ungated, and it fuses through expensive cross-attention inside the backbone. Res-Tuning shows that a tuner detached from the backbone as a parallel bypass cuts training memory by 49.7% (up to 70%) while reaching 74.10% on VTAB-1k. Its bypass operates purely in spatial and channel space, and it notes that the bypass still inherits uninformative backbone activations when the backbone lacks the needed features. We adopt its detached topology and change what the bypass sees.

Two papers that train full models from scratch support the local-global division of labor, with convolutions for high frequencies and attention for low. HLNet uses this division for restoration (29.66 dB PSNR-μ on BracketIRE+). FUSE assigns different convolution types to learnable radial frequency bands. Neither works with a frozen backbone.

Two results constrain the design. HarmoniAD documents that frequency filters catch fine structure but amplify noise, which is why the gate is required rather than optional. Few-shot medical VLMs shows that elaborate prompt methods can lose to a linear probe in the 1-shot setting (46.05% for CoOp versus 52.05% for a linear probe on NCT-CRC). This warns that our module must also be tested against the cheapest readout.

Research Program

The diagnostic comes first. Take a frozen backbone $f$ and an image $x$ with 2D Fourier transform $X$. Split the spectrum into $B$ radial bands and let $x_b$ be the image with only band $b$ retained. We fit a linear decoder $D_b$ from the frozen patch features to $x_b$ on target-domain images and define the band retention error

$$E_b = \frac{\mathbb{E}\,\|x_b - D_b(f(x))\|^2}{\mathbb{E}\,\|x_b\|^2},$$

where values near 1 mean the backbone keeps essentially nothing of band $b$. We also measure each band's task relevance, $R_b$: the drop in a reference model's accuracy when band $b$ is removed from its input. The product $E_b R_b$ is our prediction of where feature-only adapters will fall short. We compute this profile for DINOv2 on Cityscapes, ACDC-Night, Dark Zurich, Potsdam, Vaihingen and LoveDA, and for the VTAB-1k specialized and structured groups.

The adapter comes second. A tiny network $s(x)$ reads pooled spectral energy statistics of the input and outputs a nonnegative gain per band. The gated spectrum is

$$\tilde X(u) = g_\theta\big(b(u);\, s(x)\big)\, X(u),$$

where $u$ is a frequency coordinate, $b(u)$ its band, and $g_\theta \ge 0$ a real gain. Because $g_\theta$ is real, phase is preserved exactly, which avoids the phase loss in VFPT. The inverse transform of $\tilde X$ feeds a shallow depthwise convolutional pyramid at strides 8, 16 and 32. Its outputs are added, through zero-initialized projections, to frozen backbone features taken from several depths. The backbone runs once without gradients, as in the Res-Tuning bypass, so training memory scales with the side network alone.

To get around periodic-boundary artifacts, the image is mirror-padded before the transform. We add a mild sparsity penalty on $g_\theta$ so the gate admits only bands it needs. The analysis then asks whether the learned gate mass concentrates on bands with high $E_b R_b$. If it does, that agreement links the diagnostic and the method.

Baselines

The do-nothing reference is the frozen DINOv2 backbone with the same segmentation decoder trained on top and no adapter. This measures how much any adapter adds. Alongside it, we run full fine-tuning under the same schedule as the upper-cost reference.

The strongest existing competitors are NightAdapter on the night benchmarks and Earth-Adapter on remote sensing. They are the closest published methods and attack the same frozen-backbone gap with frequency decomposition. The file gives only relative improvements: NightAdapter's 13.4% mIoU over the prior method on Cityscapes→ACDC-Night and up to 5.7% on night-to-night transfer, and Earth-Adapter's 9.0% domain-adaptation average gain and +3.1% domain-generalization gain. No reported absolute mIoU was found on Lacuna for either. We will therefore re-run both from their released settings on our splits rather than quote numbers. Each will be matched to our method at roughly 10M trainable parameters, with the same decoder, data, iteration count and a hyperparameter search budget of equal size. Beating them means a mean improvement over three seeds that exceeds seed variance on the cross-domain targets, with no loss on the in-domain source. LoRA at matched parameter count is included as the standard parameter-efficient baseline. The spatial Res-Tuning bypass is included as the detached-topology control.

The key ablations isolate each ingredient. Replacing the gate with all-pass gives a ViT-Adapter-like full-band convolutional stream reading raw pixels. Freezing the gate to a static learned mask removes input conditioning. Moving the identical gate-plus-convolution module onto backbone features instead of the raw image tests the "re-supply from the input" claim directly. A variant that keeps only the real part of the spectrum tests phase. Replacing the convolutional pyramid with an MLP of equal size tests the local inductive bias.

Why not just…?

Why not just use ViT-Adapter's convolutional stem, which already reads raw pixels? This is the real threat. Its stem already restores high frequencies, so our all-pass ablation is essentially this method in a cheaper, detached form. We expect the gate to matter only where the high bands carry noise or lighting artifacts, as on ACDC-Night and Dark Zurich, and not on clean in-domain data. We will report this honestly. If the all-pass stream matches the gated one under night conditions and under synthetic sensor noise, the contribution narrows to the diagnostic and the detached, memory-cheap topology, and we will say so.

Why not just filter the backbone's features, as NightAdapter and Earth-Adapter do? If a band is absent from the features, no filter on the features can bring it back. The feature-side ablation, which uses the same module applied to features rather than the image, tests this directly. Our prediction is specific: its shortfall relative to the raw-image version should grow with $E_b R_b$, and vanish on domains where the backbone retains all relevant bands.

Why not just use LoRA or full fine-tuning? F-Adapter shows that LoRA hits a spectral error floor on broadband targets, with frequency-aware adapters cutting error by 16.7% to 25.4% relative to it. On VTAB-1k, full fine-tuning reached only 65.57% against VFPT's 73.20%. Both are still run as baselines. Full fine-tuning also serves as the check that our gains are not simply an artifact of the backbone being frozen.

Why not just train a linear probe? For dense segmentation, a probe on single-scale ViT tokens is the do-nothing reference above, and it lacks multi-scale detail. For few-shot classification, the medical VLM results make it a serious competitor. We therefore include 1-, 4- and 16-shot VTAB-1k and PatchCamelyon runs against a linear probe, and treat a loss there as a stated limitation.

Proposed Experiments

The main experiments train on Cityscapes and evaluate on ACDC-Night. Night-to-night transfer uses Dark Zurich, Nighttime Driving and BDD100K-Night. Remote-sensing domain adaptation and generalization use Potsdam, Vaihingen and LoveDA. VTAB-1k, with 1–16-shot variants, is a classification check. All runs use frozen DINOv2 and report mIoU or accuracy, trainable parameters, peak training memory and inference latency.

Four checks decide the outcome. First, the retention profile $E_b R_b$, computed before training, should rank-correlate with the gain of the raw-image module over the feature-side ablation across datasets and bands. A null correlation falsifies the rigidity-is-band-specific claim. Second, on in-domain Cityscapes, where we expect the backbone to retain the relevant bands, our module should only match feature-side adapters, not beat them. Third, under synthetic high-frequency noise added to ACDC-Night, the all-pass stream should degrade faster than the gated stream. Fourth, at matched accuracy, training memory should fall well below NightAdapter's, in the range Res-Tuning reports for detached bypasses. Passing the first two checks establishes the scientific claim. Passing the third justifies the gate over the ViT-Adapter-style alternative.