Choosing What an Adapter Sees First: Frequency-Targeted, Identity-Preserving Initialization for Restoration Adapters

When a pretrained image model is adapted with a low-rank adapter, a small trainable add-on whose output is summed onto a frozen weight, the standard recipe makes the adapter start at exactly zero. At the first training step the adapted model is then identical to the pretrained one. We propose that this first step matters for a second reason. In standard LoRA one adapter factor is random and the other is zero, so the first gradient updates are steered by whatever the loss emphasizes at step zero. For image restoration, an ordinary pixel loss puts almost all of its weight on low-frequency error such as color and smooth shading. This holds even when the real gap between the pretrained model and the new domain lies in edges and texture.

The idea is to keep the adapter's output exactly zero at initialization, but to choose its nonzero factor deliberately. That factor should span the input directions that matter most for reducing a wavelet-band-balanced loss on target data, rather than random directions. The model starts out unchanged, but its adapter is aimed from the beginning at the frequencies the pretrained model gets wrong. The central empirical question is whether this aim persists. If an adapter's subspace settles early in training, then what the loss emphasizes in the first few hundred steps shapes the final result, and initialization becomes a way to supervise frequencies. If the subspace keeps rotating throughout training, the idea fails, and we will say so.

Background and the gap

Identity-preserving initialization is widespread. LoRA sets one factor to zero so that the update vanishes at step zero. Other methods do the same thing in different places. Whisper-Flamingo gates new visual cross-attention with a scalar that starts at zero. ADEPT duplicates layers with function-preserving initialization. RidgeLoRA and CDKA keep exact identity behavior at the start while relaxing other constraints. CDKA reports that zeroing one Kronecker factor was necessary for stable training. That result constrains our design: we keep exact identity rather than trade it away.

Beyond Zero Initialization motivates the proposal from the other side. It shows that zero initialization creates an "inertial constraint": one factor receives no gradient at first, which makes training sensitive to learning rate. Its fix, Init[AB], gives up exact identity in exchange for faster convergence (a reported 2x speedup on QNLI). That work studied only language classification and never asked which signal the early gradients carry.

Frequency-domain supervision supplies the other half. WHFL reweights wavelet subbands to fight spectral bias, the tendency of networks to fit coarse low-frequency structure before fine detail. SRCliqueNet normalizes subband magnitudes ("Mode 4") so the low-frequency band does not drown out the others. Adversarial Sobolev Alignment replaces the plain squared error with a Sobolev norm, a squared error that weights high frequencies more heavily. All three train full networks. None of them involves frozen backbones or adapter initialization.

The nearest connecting work works in different spaces. Spectral Adapter tunes the singular-vector spectrum of weight matrices, which is not image frequency. CrossSpectra Fourier-transforms stacked weights across depth, keeps the low-frequency coefficients, and initializes them randomly. MultLFG and BIR-Adapter use wavelet or Sobel decompositions only at sampling time. ResAdapter trains resolution adapters with ordinary losses and ordinary LoRA initialization.

None of these papers uses output-frequency information to decide how an adapter is initialized, and none measures whether early adapter directions persist under frequency-weighted training.

Core Claim

Write an adapted layer as $h = W_0 x + BAx$. Here $W_0$ is the frozen pretrained weight, $A \in \mathbb{R}^{r \times d}$ projects the layer input $x$ to rank $r$, and $B \in \mathbb{R}^{d' \times r}$ is set to zero. The training loss is a band-balanced wavelet loss:

$$\mathcal{L}_w = \sum_{b \in \{LL, LH, HL, HH\}_{\text{scales}}} \frac{1}{E_b}\,\big\|\Psi_b(\hat y) - \Psi_b(y)\big\|_1 .$$

Here $\Psi_b$ extracts wavelet subband $b$ of the output, and $E_b$ is the mean energy of that subband in target images, in the spirit of SRCliqueNet's normalization.

We take $G_w = \partial \mathcal{L}_w / \partial W$, the gradient of this loss with respect to the frozen weight, averaged over a few hundred target-domain batches before training begins. We then set the rows of $A$ to the top-$r$ right singular vectors of $G_w$, with a fixed scale, and keep $B = 0$.

At step zero the model is exactly the pretrained model. The first update to $B$, which is $\nabla_B = (\partial\mathcal{L}/\partial h)(Ax)^\top$, then acts in the input directions that most reduce the frequency-balanced error. With a random $A$, those directions would be arbitrary. With a gradient taken from a plain pixel loss, they would be dominated by low frequencies.

The claim has two parts. The first is that this frequency-targeted identity initialization (FTI) reaches better high-frequency fidelity per step and at convergence than random-$A$ initialization trained with the same loss. The second is that the reason is subspace persistence: the column space of $A$ stays close to its initial value. The second part is what makes initialization matter at all, and we measure it directly.

Research Program

The primary testbed adapts a pretrained bicubic ×4 super-resolution backbone of EDSR type to real optical degradations. The training data is RealSR and DRealSR, plus DIV2K with synthetic blur and noise kernels. The adapters are LoRA modules on convolution and attention projections. This regime is cheap enough to run many seeds and learning rates. Identity preservation is also measurable here, as retained bicubic performance on Set5, Set14, BSDS100 and Urban100. The secondary testbed is BIR-Adapter-style cross-attention adapters on a frozen diffusion model. In that setting the wavelet loss is applied to the decoded one-step estimate of the clean image.

We will build three pieces. The first is the FTI initializer, which costs one gradient accumulation and one truncated SVD per layer. The second is band-resolved learning curves, meaning per-subband error over training steps. The third is a subspace-persistence measurement: the principal angles between the column space of $A$ (and of $B$) at step $k$ and at convergence. The persistence measurement is the mechanistic heart of the project. It tells us whether early gradient content leaves a lasting imprint, a question Beyond Zero Initialization raised implicitly but did not test for frequency content.

A secondary analysis connects to CrossSpectra's finding that about 70% of adaptation energy sits at low cross-layer frequencies. We will check whether frequency-targeted adapters break that pattern. If they do, a low-pass weight parameterization would cap high-frequency restoration.

Baselines

The naive reference point is the frozen pretrained model with no adaptation, together with standard zero-initialized LoRA (random $A$, zero $B$) trained with a plain $L_1$ pixel loss. This shows how much of the gap to the new domain an ordinary adapter closes. It also shows how badly the high-frequency bands lag.

The strongest competitor on our exact question is standard zero-initialized LoRA trained with the same band-balanced wavelet loss $\mathcal{L}_w$. It isolates whether initialization adds anything beyond the loss itself. A second strong competitor is Init[AB] from Beyond Zero Initialization, trained with $\mathcal{L}_w$. It attacks the same slow-start problem by giving up exact identity.

Two further references bound the comparison. Full-parameter fine-tuning with $\mathcal{L}_w$ is an upper reference on fidelity and a lower reference on retention. A Sobolev-loss variant following Adversarial Sobolev Alignment, without its adversarial component, checks that our results do not depend on the particular frequency weighting.

The key ablation is gradient-aligned initialization computed from the plain $L_1$ loss instead of $\mathcal{L}_w$. It separates "aiming the adapter at the target gradient" from "aiming it at the frequency-balanced gradient."

Fairness rules apply to every method. All use the same rank and parameter count, the same data order, and the same number of optimizer steps. FTI's pre-training gradient pass is charged against its step budget. Each method gets an independent learning-rate sweep over the same grid, and results are reported over three seeds.

On the reference numbers: no reported result found on Lacuna for LoRA-style adaptation of a pretrained super-resolution network to RealSR or DRealSR, and BIR-Adapter's file entry quotes no per-split scalars. The only full-training anchors are SRCliqueNet+ at 26.80 dB PSNR versus EDSR at 26.64 dB on Urban100 ×4 (SRCliqueNet) and 31.94 dB for the top NTIRE 2024 entry on the DIV2K test set (NTIRE 2024). These are context for the bicubic retention check, not targets for the adapters.

Beating the strongest baseline means two things. FTI must lower high-band wavelet error and Log-Spectral Distance on RealSR/DRealSR by more than seed variance, without losing PSNR. It must also match or exceed zero-init LoRA with $\mathcal{L}_w$ in retained bicubic PSNR on Urban100. A gain that appears only at early steps and vanishes at convergence would count as faster training, not a better result, and we will report it that way.

Why not just…?

Why not just train ordinary zero-initialized LoRA with the wavelet loss and stop there? This is the real threat. If adapter subspaces keep rotating throughout training, the loss alone will eventually steer the adapter wherever FTI would have, and FTI only saves a few steps. The subspace-persistence measurement settles this. If the principal angles between early and final subspaces are large for random-$A$ LoRA, and the two methods converge to the same band errors, the initialization claim is false and we will report it as a negative result about adapter dynamics.

Why not just use non-zero Init[AB] to escape the inertial constraint? Beyond Zero Initialization shows it speeds convergence in language tasks. But it injects random output noise at step zero, which in restoration shows up as immediate artifacts and lost pretrained fidelity. It also does nothing to choose which frequencies the adapter targets. Our retention metric on the original bicubic task, measured at step zero and throughout training, makes that cost visible. The band-resolved curves show whether Init[AB]'s speedup reaches the high-frequency bands at all.

Why not just schedule the loss, weighting high frequencies heavily for the first few hundred steps with a random $A$? If early steps imprint the subspace, a high-frequency warm-up might achieve the same aim without any SVD. We run this curriculum as an explicit baseline at matched steps. If it matches FTI, the honest conclusion is that what matters is the spectral content of early gradients, not the initializer. We would then present the curriculum as the simpler method, with the persistence analysis as the explanation.

Why not just add a perceptual (VGG) loss? SPSR and Adversarial Sobolev Alignment both report that perceptual objectives produce plausible but geometrically wrong texture. That is the opposite of what retention-constrained adaptation needs. We include LoRA with $L_1$ plus perceptual loss and score it on Log-Spectral Distance and Urban100 edge fidelity, not only LPIPS.

Proposed Experiments

Each method is trained on RealSR, DRealSR and DIV2K with synthetic degradations, at ranks 4, 16 and 64. The evaluation reports PSNR, SSIM, LPIPS, Log-Spectral Distance and per-subband wavelet error on the real-degradation test splits, plus retained bicubic PSNR and SSIM on Set5, Set14, BSDS100 and Urban100.

The first decisive check is persistence. We plot the principal angles between the adapter subspace at steps 0, 100, 1000 and final, for random-$A$ versus FTI. If those angles are small and FTI's high-band advantage persists at convergence, the claim stands. If the angles are large and the advantage disappears, it falls.

The second decisive check is learning-rate robustness. We repeat the sweep at learning rates both lower and higher than the tuned optimum, the regime where Beyond Zero Initialization found zero initialization fragile. This tests whether a targeted $A$ removes that fragility while keeping identity.

The third check is transfer to the diffusion testbed. We compare BIR-Adapter-style adapters initialized randomly and with FTI, under the same wavelet loss on the decoded estimate, and measure hallucination on DIV2K degradations with Log-Spectral Distance and edge-map agreement. This shows whether the effect survives a much larger frozen model with cross-attention insertion points.