Where Low-Rank Adapters Lose High Frequencies

Big image models are usually specialized to a new task through parameter-efficient fine-tuning. Instead of updating every weight, which is expensive in memory and storage, one trains a small add-on. The most common add-on is LoRA, a pair of thin matrices whose product is added to each frozen weight. A separate line of work shows that image restoration networks produce blurry outputs because pixel losses mostly reward getting the smooth, low-frequency content right. Supervising the output in the Fourier or wavelet domain fixes much of this. This note asks what happens when the two are combined. Does a low-rank adapter have the capacity to learn the high-frequency corrections that a frequency loss asks for? Or does restricting the update to a few directions quietly re-impose the blur the loss was meant to remove?

Our claim is that the gap between parameter-efficient and full fine-tuning on restoration tasks is not uniform across frequencies. It is concentrated in the high-frequency bands of the output. It grows as the adapter budget shrinks, and adding a frequency loss narrows it only partly, because under a uniform rank the high-frequency gradient signal is spread across layers the adapter does not cover well. We propose to measure this gap band by band. We then test one targeted fix: split the loss gradient by output frequency band, and use the high-band part to decide where adapter rank goes and how it is initialized. If the fix does not beat the obvious combination of standard LoRA plus a frequency loss at equal budget, the band-resolved diagnosis is still a useful negative result.

What exists and what does not

Frequency supervision supplies the loss we reuse. Focal Frequency Loss for Generative Models weights Fourier-domain errors toward frequencies the model finds hard, cutting CelebA VAE reconstruction FID from 97.0 to 83.8. Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution predicts Haar wavelet subbands directly, and shows the low-frequency subband dominates the gradient unless it is renormalized. Frequency-Supervised MR-to-CT Image Synthesis supervises a high-pass residual, lowering MAE from 79.09 to 72.71. All three train full models. None asks how a restricted update space interacts with the loss.

Spectral parameter-efficient methods move the adapter into a frequency basis, but in weight space, not output space. Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps trains sparse Fourier or wavelet coefficients of the LoRA factors. Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform trains a few high-energy DCT coefficients of the weight update. CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning truncates high frequencies of the update across depth, arguing that they are mostly noise.

All three are evaluated on language and classification tasks with ordinary losses. They motivate one of our questions: does weight-space low-pass truncation also remove output-space high frequencies? The two notions of frequency are different, and nothing in the file shows they are linked. We test the link rather than assume it.

The closest vision work splits adapters by loss type. Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach (PiSA-SR) trains one LoRA on an ℓ2 pixel loss and one on perceptual losses (LPIPS and classifier score distillation) inside Stable Diffusion 2.1. Latent Harmony adapts a frozen autoencoder with encoder and decoder LoRAs and uses a high-frequency alignment loss as a stabilizer. Neither measures per-band error against full fine-tuning, and neither uses band-split gradients to shape the adapter.

The idea of "one LoRA per frequency band" is the obvious move here and is close to PiSA-SR's design. We therefore treat it as a baseline, not as the contribution.

Core Claim

Consider a frozen model adapted with total adapter budget $P$, where $P$ is the number of trainable parameters. Define the band recovery ratio as

$$\rho_b(P) = \frac{E_b^{\text{frozen}} - E_b^{\text{PEFT}}(P)}{E_b^{\text{frozen}} - E_b^{\text{full}}}.$$

Here $E_b$ is the mean squared error of the output's Fourier coefficients in radial band $b$, measured on held-out data. "Frozen" is the unadapted model, and "full" is full fine-tuning with the same loss.

We predict three things. First, $\rho_b$ falls with frequency: adapters recover low bands almost fully and high bands poorly. Second, the drop is steeper at small $P$ and for adapters truncated to low weight-space frequencies (DCT or CrossSpectra style). Third, choosing layer ranks and initial directions from the high-band component of the gradient raises high-band $\rho_b$ at fixed $P$, beyond what adding a frequency loss to uniform LoRA achieves.

Research Program

The first step is diagnosis. For each task we run full fine-tuning and a sweep of adapters: LoRA at several ranks, the selective-DCT adapter, a CrossSpectra-style depth-truncated adapter, and 1LoRA. 1LoRA sums input features into a single learned vector per layer, and may therefore act as a spatial low-pass on activations. Each adapter is trained with two losses: ℓ1 alone, and ℓ1 plus focal frequency loss. We report $\rho_b$ over eight radial bands, together with peak memory and trainable-parameter counts. This alone tests Gap 2 from the file.

The second step is the method, which we call band-gradient allocation. Before adapter training, we run a short probe of a few hundred batches. The probe computes weight gradients layer by layer, so peak memory stays near that of one layer's full gradient. The output error is split into a low band and a high band by a fixed radial Fourier mask, and each band's loss is backpropagated separately. For each layer we take the top singular directions of the high-band gradient that are not already spanned by the top low-band directions. We allocate rank across layers in proportion to that residual energy, and initialize each LoRA's input factor from those directions. Training then proceeds as ordinary LoRA with ℓ1 plus frequency loss, and the adapter merges into the weights at inference exactly as standard LoRA does.

The probe cost is reported and charged against the method in every comparison.

Proposed Experiments

The controlled setting takes a pretrained convolutional super-resolution network, trained on bicubic DIV2K, and adapts it to 4× real-world degradations. It is evaluated on Urban100, whose building textures are rich in high frequencies. The file reports 26.64 dB PSNR for EDSR and 26.80 dB for SRCliqueNet+ on Urban100 4× bicubic (Joint Sub-bands Learning). These are reference points for the base model's quality, not direct competitors under our degradation shift.

The large-model setting follows PiSA-SR: one-step Stable Diffusion 2.1 super-resolution, trained on LSDIR and FFHQ and evaluated on RealSR and DrealSR with PSNR, SSIM, LPIPS and band-resolved error. No reported result was found on Lacuna for PiSA-SR's quality metrics on RealSR or DrealSR. The file gives only its 0.09 s single-step latency, so we rerun PiSA-SR ourselves.

The medical setting adapts a reconstruction model to FastMRI knee at 8× acceleration, where missing high frequencies are clinically meaningful. There, MDPG reports 30.43 dB PSNR and 0.740 SSIM, the strongest number in the file for that task. We first reproduce focal frequency loss on the CelebA VAE, checking against the 97.0 to 83.8 FID drop, so that loss implementation errors cannot masquerade as findings.

Baselines

The naive reference is the frozen pretrained model with no adaptation. It anchors $\rho_b = 0$. The upper reference is full fine-tuning with ℓ1 plus focal frequency loss, which anchors $\rho_b = 1$ and whose memory cost is the overhead the adapters exist to avoid.

The strongest direct competitor is standard uniform-rank LoRA (LoRA) trained with ℓ1 plus focal frequency loss. It is the obvious combination of the two directions, and it is what a practitioner would try first. It gets the same data, the same step count, the same learning-rate search of equal size, and a rank chosen to exactly match our total parameter budget. It also receives extra training steps equal to our probe's compute, so it does not lose on budget.

The second competitor is a PiSA-style dual LoRA. Here one adapter is trained on low-band loss and one on high-band wavelet-subband loss, at the same total budget. This tests whether explicit band-separate adapters suffice. On the diffusion task we also run PiSA-SR's original pixel/semantic split.

Beating these means higher high-band $\rho_b$ and higher PSNR or lower LPIPS at matched parameters and matched total compute, across at least three seeds, with confidence intervals that do not overlap.

Two ablations isolate the frequency ingredient. The first allocates rank from the total, unsplit gradient. The second keeps band-split allocation but uses random initialization. If total-gradient allocation matches our method, the frequency split is not doing the work, and we will say so.

Why not just…?

Why not just add a frequency loss to ordinary LoRA? This is the real threat. If a frequency loss alone lifts high-band recovery to near full fine-tuning, the allocation step is unnecessary. Our experiments settle it directly, because this is the main baseline. The diagnostic sweep still reports whether the high-band gap exists at all, and how it depends on rank.

Why not just raise the rank? Higher rank buys capacity everywhere, but SeLoRA finds LoRA matrices already carry redundant density, which suggests uniform capacity is poorly spent. We plot $\rho_b$ against $P$ for uniform LoRA. If our method at budget $P$ matches uniform LoRA at, say, $2P$, that is the concrete saving. If no rank closes the high-band gap, the problem is not capacity alone.

Why not just fine-tune everything? Full fine-tuning is the ceiling, and UltraHR-100K used full post-training with frequency losses while naming 4K training cost as an open bottleneck. We report peak memory for every method. The question is how much of full tuning's high-band quality survives at a small fraction of its memory.

Why not use a spatial edge loss instead of Fourier bands? A Sobel or Gaussian-difference loss would also emphasize edges, as the high-pass residual in Frequency-Supervised MR-to-CT Image Synthesis does. We include a Gaussian-difference variant of the gradient split. Allocation needs only a low/high split, so if the spatial filter works equally well we adopt it as the simpler choice.

Decisive checks

The proposal lives or dies on two results. First, $\rho_b$ for uniform LoRA with a frequency loss must fall measurably in the top bands on at least two of the three tasks. Otherwise there is no gap to fix, and the paper becomes a negative result about Gap 2. Second, band-gradient allocation must beat both the uniform baseline and the total-gradient ablation on high-band error at matched compute.

A third, independent check asks whether adapters truncated to low weight-space frequencies show a worse high-band output gap than plain LoRA at equal budget. Either answer is informative for how methods like CrossSpectra should be used on dense prediction tasks.