Frequency-Gated LoRA: Letting Adapters Treat Coarse and Fine Image Structure Differently

Low-rank adaptation (LoRA) fine-tunes a large frozen model by adding a small trainable update $BA$ to each weight matrix. It is cheap, but it is fragile: move the learning rate a little and accuracy can collapse, so every new task needs a careful search. This note starts from an observation about vision transformers that makes the fragility easier to reason about. A LoRA update is applied to each image patch token separately. Because of that, it commutes with a Fourier transform taken over the grid of patches. In plain terms, the adapter applies exactly the same change to the coarse layout of an image as to its fine texture and noise. It cannot decide to adjust one without adjusting the other.

We propose frequency-gated LoRA. Before the low-rank update, the patch-token grid is split into a few spatial-frequency bands, from coarse to fine. All bands share the same low-rank matrices. A small bounded gate then decides, for each input and each band, how strongly each rank component of the update is applied. This is adaptive feature fusion, the familiar practice of mixing several feature streams with input-dependent weights, applied inside the adapter to frequency bands. Our central hypothesis is that at high learning rates LoRA damages the pretrained model mostly through the fine-scale bands, by fitting texture idiosyncrasies of small training sets. If that is right, a gate that can close those bands should widen the range of learning rates that work, without the capacity limit that comes from bounding the whole update. The hypothesis can be checked directly by measuring where the adapter's changes to the pretrained features concentrate, and we treat that measurement as the first result, not an afterthought.

Core Claim

Let $X \in \mathbb{R}^{N \times k}$ be the $N$ patch tokens (width $k$) entering an adapted linear layer, arranged on an $h \times w$ grid. Let $X_b = \mathcal{F}^{-1}(M_b \odot \mathcal{F}(X))$ be the component of $X$ in radial frequency band $b$, where $\mathcal{F}$ is the 2D Fourier transform over the grid and $M_b$ is a fixed radial mask, chosen so that $\sum_b X_b = X$. The adapted layer computes

$$ H = W_0 X + \frac{\alpha}{r} \sum_{b=1}^{K} B \,\mathrm{diag}\big(g_b(X)\big)\, A\, X_b , $$

where $W_0$ is the frozen weight, $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ are the usual shared LoRA factors, $K$ is the number of bands (3 or 4), and $g_b(X) \in (0,1)^r$ is a sigmoid-bounded gain vector. That gain vector is produced by a tiny linear map from the per-band log-energies of $X$. If all $g_b$ are equal, this is exactly LoRA with a per-component rescaling. The only new freedom is frequency-dependent gain, and it costs $K \cdot r$ gain outputs plus a small gate, alongside two FFTs over a grid that is only 14×14 for a ViT-B/16. The class token bypasses the band split.

The claim has two parts, and each can fail on its own. The first is diagnostic: under standard LoRA, the relative feature change $\|\Delta W X_b\| / \|W_0 X_b\|$ grows fastest in the high bands as the learning rate increases, and that growth predicts accuracy collapse. The second is interventional: with the gate, the region of learning rates within one point of the best accuracy becomes wider than LoRA's, and at least as wide as that of norm-bounded methods. Accuracy at the best learning rate should not fall below theirs.

Relation to Prior Work

Frequency-domain adapters supply the mechanism but were built for other goals. FouRA learns low-rank projections over spectral representations, using an MLP-generated input-dependent mask, to prevent output collapse in diffusion fine-tuning. LoCA learns which DCT coefficients of the weight update to tune. Selective DCT fine-tuning picks high-energy DCT coefficients per band to save memory. F-Adapter allocates more adapter width to low-frequency shells of neural PDE operators using fixed energy priors, and its ablation shows frequency allocation beats unweighted adapters by 3.0 to 5.2%. None of these four measures learning-rate sensitivity. Three of them take the transform of the weight matrix or the channel axis, not the spatial token grid where the commutation argument applies. F-Adapter's allocation is fixed rather than input-dependent. Our gain is data-dependent, bounded, and evaluated specifically as a stabilizer.

Stabilization work defines the competition. DeLoRA separates the direction of the update from a single learnable strength and keeps it within a ball scaled to the weight norm. ETHER uses norm-preserving hyperplane reflections, which are stable across learning rates spanning orders of magnitude but acknowledge an expressivity ceiling. Both bound the entire update uniformly. Our bet is that bounding per band is enough, because the harm is concentrated there, and that it costs less expressivity. LoFT and AltLoRA instead attribute fragility to optimizer momentum that goes stale as the low-rank subspace rotates. That is an independent explanation. Our design keeps $A$ and $B$ shared and fixed in shape, so their optimizer fixes can be combined with it. We test whether the two effects add or overlap.

Fusion work motivates the gate. LoRA-Flow shows that token-dependent gating between frozen LoRAs beats static weights, reaching 37.6% on MGSM against 28.7% for LoRA-Hub. MultLFG argues that interference between adapters is lower in wavelet subbands, though only for training-free composition. Attentive layer fusion shows that input-dependent weighting of frozen layers recovers information that linear probes miss. We move that gating inside a single adapter that is being trained, and apply it across frequency bands rather than across adapters or layers. HarmoniAD constrains the design: it reports that structure-oriented high-frequency filters are noise-sensitive. That supports letting the gate close high bands. FUSE cuts the other way, reporting that models over-rely on low-frequency cues. This is why we do not hard-code a low-pass prior and instead learn gains per input.

Research Program

The work proceeds in three stages. First, we instrument standard LoRA on a ViT-B/16 pretrained on ImageNet-21K. We sweep learning rate over a fixed log grid of seven values and record per-band relative feature drift in every adapted layer throughout training, on a subset of VTAB-1k tasks from each of its three groups (natural, specialized, structured). If collapse is not preceded by disproportionate high-band drift, the core hypothesis is wrong. We would then report that negative result and not proceed to method claims. Second, we implement frequency-gated LoRA on the same query, value, and MLP layers, at matched rank and with parameter counts within 1% of LoRA. We run the full sweep on all 19 VTAB-1k tasks with three seeds. Third, we test whether the effect carries to Stable Diffusion v1.5 DreamBooth, where UNet features are also spatial grids and DeLoRA and ETHER have published results. Here, fragility shows up as lost subject fidelity or lost diversity, not as accuracy collapse.

Sensitivity is measured three ways. The first is the width, in log-learning-rate units, of the interval within one point of each method's best accuracy. The second is mean accuracy over the whole grid. The third is the accuracy obtained when a single learning rate, chosen on held-out tasks, is used for every VTAB task. The third corresponds to the practical question of whether the search can be skipped.

Baselines

The naive reference points are two. One is linear probing of the frozen backbone. The other is the strongest frozen-backbone probe in the file: ComBo reports 74.6% average VTAB-1k accuracy probing a single ImageNet-21K ViT. Any fine-tuning method that fails to beat that number at most learning rates has no case for touching the backbone.

Standard LoRA is the direct baseline. It is run at the same rank, on the same layers, with the same seven-point learning-rate grid, seeds, schedule, and step count. Because both methods get the same search budget, the comparison is of sensitivity, not of how much tuning each received.

The strongest competitors on the stability question are DeLoRA and ETHER. Both were designed exactly to widen the usable learning-rate range. DeLoRA reports training stable at learning rates 4× to 8× above LoRA's collapse threshold. ETHER reports stability across multi-order-of-magnitude sweeps and was evaluated on VTAB-1k, but the file gives no VTAB number for it. No reported VTAB-1k result for DeLoRA was found on Lacuna either. We therefore rerun both under our grid. On DreamBooth, the reference numbers are DeLoRA's DINO 0.693 and CLIP-I 0.820 from DeLoRA. Beating them means one of two outcomes at matched parameters. Either the gated method has a wider within-one-point learning-rate interval than DeLoRA with best accuracy no more than 0.3 points lower, or it has equal width with higher best and mean-over-grid accuracy.

LoFT-simple represents the optimizer-side explanation and is run with the same grid. FouRA represents prior frequency-domain LoRA with input-dependent gating. It checks whether any spectral parameterization gives the robustness, or only one that follows the token grid.

Three ablations isolate the key ingredients. The first uses the band split with fixed, equal, ungated gains. It is mathematically LoRA plus a per-component rescaling, and it tests whether the gate matters. The second uses a gate driven by overall input energy without any band split, which tests whether frequency matters. The third replaces input-dependent gains with static learned per-band gains, which tests whether the adaptive-fusion element matters beyond a fixed spectral preference.

Why not just…?

Why not just tune LoRA's learning rate more carefully, or raise the rank? Prior evidence cited in the file indicates that beyond a minimal rank, learning rate matters more than rank. LoFT shows a rank-8 adapter with proper optimization matching LoRA at rank 64 on HumanEval, so extra rank does not buy robustness. Careful tuning does work, but it is the cost we want to remove. The single-learning-rate-across-tasks measurement shows directly how much accuracy is lost by skipping the search under each method.

Why not just use DeLoRA or ETHER? This is the real threat. A global norm bound may flatten the learning-rate curve as well as band gates do. If our sweep shows DeLoRA matching both width and best accuracy, the frequency ingredient is not earning its FFTs. The honest conclusion would then be that fragility in ViTs is a magnitude problem, not a frequency problem. The remaining question would be whether band gains added on top of DeLoRA recover the expressivity that ETHER admits it lacks, and we run that combination as a secondary arm.

Why not just low-pass the adapter's input, or give each band its own tuned learning rate? Both are static versions of the same idea and cheaper. The fixed per-band gain ablation covers the first. For the second, we give per-band learning rates a K-times larger search budget. If they match the gate only at that cost, the gate's value is in avoiding the search. If they match at equal budget, the adaptive part of the proposal is unnecessary and we will say so. FUSE's finding that models over-rely on low-frequency cues is the reason we expect a hard low-pass to hurt structured VTAB tasks.

Why not just fix the optimizer, as LoFT and AltLoRA do? Stale momentum is a competing explanation of the same symptom. The drift diagnostic run under LoFT-simple distinguishes the two. If LoFT removes the high-band drift spike, the frequency story is downstream of the optimizer story. If the spike persists, the explanations are separate and the methods should stack.

Decisive Checks

Everything rests on the first-stage measurement: whether LoRA's collapse at high learning rates is preceded by drift concentrated in fine-scale bands of the patch grid. After that, two comparisons settle the proposal. The first is against DeLoRA on learning-rate interval width at equal best accuracy across the 19 VTAB-1k tasks. The second is against the ungated band-split ablation, which shows whether input-dependent fusion of bands, not merely splitting them, produces the gain. A positive DreamBooth result at or above DINO 0.693 over a wider learning-rate range would show that the mechanism is not specific to classification.