Parameter-efficient fine-tuning freezes a large pretrained model and trains a small add-on. It is often credited with a second benefit beyond saving memory: it acts as a regularizer that keeps the model from overfitting the quirks of its training domain. Nobody has pinned down what the add-on is prevented from learning, and so what actually does the regularizing. This proposal tests one concrete answer for vision models under domain shift. The small update should be allowed to change only the coarse, low-spatial-frequency content of the image representation. It should be barred from amplifying fine texture, which is the signal that differs most between photos, sketches, and paintings of the same object.
The mechanism is deliberately minimal. Take the standard low-rank adapter (LoRA) inside a frozen Vision Transformer and feed it a spatially smoothed copy of the patch tokens instead of the raw tokens. The smoothing is a soft low-pass filter over the 2D grid of image patches, and its cutoff is learned per layer. Everything else matches LoRA, including the parameter count, up to one scalar per layer. If this beats LoRA and the existing "spectral" adapters on out-of-domain accuracy, the result locates the regularizing effect of parameter efficiency in which input-signal frequencies the update can reach, not in its rank.
Core Claim and How It Differs From Existing Spectral Adapters
A growing family of methods already fine-tunes "in the frequency domain," but that frequency is taken over the wrong axis for this question. FourierFT writes the weight update as the inverse Fourier transform of a few randomly placed coefficients. MaCP uses the discrete cosine transform (DCT), a real-valued relative of the Fourier transform, with three hand-set frequency tiers. LoCA also learns where the DCT coefficients sit, using finite-difference steps. CrossSpectra takes a 3D Fourier transform over stacked attention updates across layers. All four apply the transform to the weight matrix, whose row and column indices are neuron identities with no spatial order. A "low frequency" there means smoothness across arbitrarily numbered neurons. That buys compression, and these papers report large parameter savings, but it has no connection to image texture versus image shape. None of them was evaluated under domain shift.
FouRA applies low-rank projections to features in a Fourier or DCT basis and gates frequency components with an input-dependent network. It targets diversity collapse in diffusion fine-tuning, not robustness to new domains.
Frequency along the image axes is where domain style lives. Deep Frequency Filtering supplies the motivating evidence. It learns instance-adaptive masks on the 2D Fourier spectrum of intermediate CNN feature maps, reports 67.6% on Office-Home, and argues that networks exploit high-frequency domain-specific texture. That work trains the whole network with an added filtering branch. It is not an adapter on a frozen foundation model, and it does not ask how filtering interacts with a parameter budget.
On the parameter-efficiency side, Mixture-of-Adapters shows that adapters on large pretrained models produce flatter loss surfaces and strong domain generalization. It routes tokens among adapters of different ranks and has no frequency structure.
The proposal sits between these. It keeps the frozen-backbone, low-parameter setting of the second group and imposes the image-frequency restriction of the first. It is also a direct probe of the question the literature leaves open. LoCA argues that full fine-tuning updates are not low-rank, while LoRA's premise is that they are. Is the useful constraint low rank, or band limitation?
The Adapter
Let $Z \in \mathbb{R}^{h \times w \times d}$ be the patch tokens of one block, arranged on their $h \times w$ spatial grid with $d$ channels. Let $\mathcal{D}$ be the 2D DCT over the two grid axes, applied separately to each channel. Let $\rho(u,v)$ be the normalized radial frequency of DCT coordinate $(u,v)$. The layer's soft cutoff is $c$ and a fixed temperature is $\tau$. A frozen projection $W$ becomes
$$ W Z + \big(\mathcal{D}^{-1}(m_c \odot \mathcal{D} Z)\big) A B, \qquad m_c(u,v) = \sigma\!\left(\frac{c - \rho(u,v)}{\tau}\right), $$
where $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times d}$ are the usual rank-$r$ LoRA factors, $B$ is initialized to zero, and $\sigma$ is the logistic function.
The spatial filter and the channel mixing act on different axes, so they commute. The adapter is therefore exactly LoRA applied to a low-passed copy of the input, and it can be implemented that way. As $c \to \infty$ it reduces to LoRA, which makes the key ingredient cleanly separable. The cutoff is continuous and trained by ordinary backpropagation in one pass, following the differentiable cutoffs and sigmoid gates surveyed in Direction 1. This avoids LoCA's alternating finite-difference scheme. Because $c$ is learned separately in each layer and in both attention and feed-forward projections, the learned cutoffs directly show how frequency capacity is allocated across depth. CrossSpectra, restricted to attention, and F-Adapter (F-Adapter), with capacity hard-coded to physical energy spectra, leave that question open.
The DCT runs on a 14×14 grid for a ViT-B/16 at 224 pixels, so its added cost is negligible next to attention. The cost the method does add is that the update mixes tokens and so cannot be merged into $W$ after training. This is the same limitation the file notes for F-Adapter.
Research Program
The work has three parts.
The first is a diagnostic that sharpens the motivation. For FourierFT and MaCP on a ViT, apply a fixed random permutation to the rows and columns of each weight matrix before the spectral transform, and undo it afterward. If downstream and out-of-domain accuracy do not change, the "frequency" in weight-space methods is only a choice of basis, not a statement about signal content. That would justify moving the frequency axis to the image grid.
The second is the main comparison on multi-domain image classification, under equal parameter budgets.
The third is an analysis of what the update learned: its output energy by spatial-frequency band, the learned cutoff at each depth, the top Hessian eigenvalue following the Mixture-of-Adapters protocol, and accuracy under added high-frequency noise.
An optional variant makes $c$ depend on the input through a pooled-feature linear head, in the spirit of Deep Frequency Filtering and FouRA. Direction 1 itself flags that instance-conditioned filters can overfit source-domain style. Whether instance adaptivity helps is therefore treated as a hypothesis to test, not a default.
One cited result cuts against the hypothesis and has to be faced directly. FUSE reports that re-identification models over-rely on low-frequency cues and that mid and high frequencies carry identity. The claim here is therefore task-dependent. Band limitation should help when labels are defined by shape, and it may hurt where fine detail is the label, as with small animals in TerraIncognita. The learned cutoff is how the method is meant to adapt to that. A per-dataset breakdown will show whether it does, or whether low-pass restriction simply fails on fine-grained domains.
Baselines
All methods share a frozen CLIP ViT-B/16. They are run in the DomainBed leave-one-domain-out protocol, using the training and model-selection scripts from the DomainBed-based DiWA code. Every method gets the same number of hyperparameter trials and three seeds. The trainable parameter counts are matched to within 5% by adjusting rank or coefficient count.
The naive reference points are linear probing and zero-shot prediction with the frozen backbone, and full fine-tuning under empirical risk minimization (ERM). The file reports ERM at 78.1% on OfficeHome in the Mixture-of-Adapters paper. Plain LoRA at the same rank and budget is the parent method. Beating it isolates the band-limiting ingredient, and its weight decay and rank get the same search budget as the proposed adapter's cutoff initialization.
The weight-space spectral methods, FourierFT and MaCP, are run at the same parameter count. They are the most direct "frequency plus efficiency" competitors, and they have never been tested out of domain.
The strongest existing method is Mixture-of-Adapters. The file reports 90.6% on OfficeHome for its KAdaptation-MoA variant. I will rerun it on our backbone rather than copy the number, because its backbone may differ. On a matched backbone, beating it means exceeding its held-out-domain average by more than the seed standard deviation, with no more trainable parameters. Merely matching it would still be informative, because our adapter is a one-parameter-per-layer change to LoRA while MoA is a routed mixture.
A Deep Frequency Filtering–style baseline puts an instance-adaptive spectral mask on frozen-backbone features alongside LoRA. This tests whether filtering the features themselves, rather than constraining the update, is what matters.
The ablations isolate the key ingredient. The first fixes the cutoff instead of learning it. The second uses a high-pass mask; it has the same parameters and should do worse if the hypothesis holds. The third shuffles the token grid before filtering. This keeps an identical smoothing operator but destroys spatial meaning, so if it matches the real low-pass, the gain is generic averaging noise reduction rather than frequency selectivity.
Why not just…?
Why not just tune LoRA's rank and weight decay? This is the $c \to \infty$ limit of the proposal, so the comparison is exact rather than rhetorical. Evidence in the file that rank is not the whole story comes from other regimes. Spectral Adapter beats LoRA on GLUE (88.03 vs 86.47) by reallocating rank, and LoCA argues that full-tuning updates are not low-rank. Neither result concerns domain shift. If a well-tuned LoRA matches the band-limited adapter on held-out domains, the hypothesis is refuted, and the note will report it that way.
Why not just use Mixture-of-Adapters? It is a real threat and the strongest number in the file. Its routing works on token coordinates, though, and nothing stops its experts from fitting high-frequency texture. The frequency-energy analysis will measure whether its learned update concentrates in high bands. The two ideas are also compatible. If MoA wins outright, a band-limited MoA, with each expert fed a filtered input, is the natural follow-up, and it is cheap to run inside the same codebase.
Why not just use the existing spectral adapters, FourierFT or MaCP? Their frequency axis runs over neuron indices, so it cannot express "leave the texture alone." The permutation diagnostic tests this directly. If permuting neurons does not hurt them, their spectral structure has no image meaning. If they nonetheless match our method out of domain, compression alone is doing the regularizing, and the band-limitation story is wrong.
Why not just augment the training images in the frequency domain, for example by swapping amplitude spectra across domains? No reported result for such augmentation on these benchmarks was found on Lacuna. It is cheap, and it is a fair competitor that I will add at equal training compute. It is also complementary rather than exclusive: augmentation changes the data, while the proposal changes what the update can represent. The informative outcome is whether the two gains stack.
Proposed Experiments
The main benchmark is DomainBed: PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet. The primary metric is held-out-domain accuracy averaged over domains, reported alongside in-domain accuracy to expose any trade-off. The gradient-surgery DG code serves as a second check of the protocol on PACS, VLCS, and Office-Home.
A secondary transfer test adds the adapter to a transformer segmentation backbone for GTAV-to-real segmentation, using DAFormer's codebase. There the strongest reported figure in the file is DeFT at 55.49% average mIoU with MiT-B5. That comparison is harder to make fair, because DeFT trains decoupled full pathways. It is reported as context, not as a head-to-head.
The decisive checks are as follows. The band-limited adapter should beat budget-matched LoRA and FourierFT or MaCP on average held-out accuracy. It should beat its own high-pass and shuffled-grid ablations. Its learned update should carry measurably less high-frequency energy than LoRA's, with a lower top Hessian eigenvalue.
The learned cutoffs should vary with depth in a consistent direction across datasets, and should be higher on TerraIncognita and DomainNet than on PACS. If the shuffled-grid control matches the real filter, the frequency explanation fails. If gains appear only on PACS-like style shifts and reverse on fine-grained domains, the claim narrows to "band limitation regularizes against style shift." That is still a useful, falsifiable account of what parameter-efficient regularization is doing.
References
- Parameter-Efficient Fine-Tuning with Discrete Fourier Transform
- MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
- LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning
- CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning
- FouRA: Fourier Low-Rank Adaptation
- Deep Frequency Filtering for Domain Generalization
- Domain Generalization Using Large Pretrained Models With Mixture-of-Adapters
- F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
- FUSE: Frequency-domain Unification and Spectral Energy Alignment for Multi-modal Object Re-Identification
- Spectral Adapter: Fine-Tuning in Spectral Space
- Decoupled Finetuning for Domain Generalizable Semantic Segmentation