When a large pretrained model is fine-tuned cheaply, there are two common ways to do it. Low-rank adaptation (LoRA) learns a small additive correction, the product of two thin matrices, to each frozen weight matrix. Normalization tuning instead retrains only the per-channel scale and shift inside the model's existing normalization layers. These are usually treated as competing recipes, or are switched on together everywhere as a matter of habit. This note argues that they express two different kinds of weight change. Changing a normalization layer's scale is mathematically the same as rescaling the input columns of every weight matrix that reads from it. That update is full-rank but tightly structured. LoRA adds a few arbitrary new directions. Work on normalization has repeatedly found that shallow layers mostly need their feature statistics corrected (style, lighting, texture), while deep layers carry the semantics. This suggests a split. Shallow layers should be adapted through normalization scales, deep layers through low-rank updates, and the boundary between the two should be measured rather than guessed.
The proposal has two parts. The first is a diagnostic. For each layer, decompose the update produced by full fine-tuning into a column-rescaling part and a low-rank remainder, which gives a depth profile of what kind of change each layer needs. The second is a method, Depth-Split Adaptation. Each layer's parameter budget is allocated between normalization-scale tuning and LoRA rank according to that profile. The method is tested where the distinction should matter most: transfer under distribution shift, on VTAB-1k and on cross-manipulation face-forgery detection. A smaller check on a multimodal LLM tests whether the pattern carries over to language decoders.
Core Claim
Consider a transformer block with normalization applied before the attention and MLP layers. A weight matrix $W_0$ reads the normalized input $g \odot \hat{x} + b$, where $\hat{x}$ is the standardized activation and $g, b$ are the normalization layer's learned per-channel scale and shift. Perturbing the scale by $\delta$ and the shift by $\beta$ is exactly equivalent to
$$\Delta W = W_0\,\mathrm{diag}(\delta), \qquad \Delta \text{bias} = W_0\,\beta,$$
and the same $\delta$ is shared by every matrix fed by that normalization layer: the query, key and value projections, or the first MLP projection. This update is generically full-rank, so a rank-8 LoRA cannot represent it. In the other direction, column rescaling of $W_0$ cannot create directions that $W_0$ lacks, so LoRA can do things normalization tuning cannot. The two update families are complementary.
The claim has three parts. First, the update produced by full fine-tuning is dominated by the column-rescaling component in shallow layers and by the low-rank remainder in deep layers. Second, this boundary moves toward the input when the target task is mostly a style or domain shift rather than a semantic one. Third, putting LoRA only above the boundary and normalization tuning only below it matches or beats adapting both everywhere at equal or smaller parameter count. We expect the advantage to show most in out-of-distribution metrics, because shallow LoRA capacity is where spurious, domain-specific artifacts get memorized.
Relation to Prior Work
Two papers motivate the claim by showing that normalization tuning alone is surprisingly strong. Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models tunes only the normalization scale and shift (0.03–0.04% of parameters). It beats BitFit on language-understanding tasks, and it combines well with prefix-tuning. Tuning LayerNorm in Attention treats turning a text LLM into a multimodal one as a domain shift. In that setting, tuning only the attention-block normalization layers beats LoRA by over 20% on average across five multimodal benchmarks at 13B scale. Neither paper combines normalization tuning with LoRA in a depth-aware way. Neither offers an account of when the scale update is the right kind of update. The weight-space equivalence above supplies that account.
The LN-tuning paper also reports two findings that constrain the design. Normalization updates are largest near the output, and normalization tuning conflicts with tuning the MLP adapters. The first finding appears to contradict the hypothesis, so it must be addressed directly. Large output-side scale changes on in-distribution language tasks are compatible with our view if those changes are cheap substitutes for semantic re-weighting. The diagnostic will tell us whether, in deep layers, the scale component is large in absolute terms but small relative to the low-rank remainder. The conflict finding is something the decomposition can explain: an adapter on the MLP and a scale change on the normalization layer feeding it compete to express the same rescaling. We will test this by measuring how much of each module's update overlaps the other's span.
The domain-shift literature supplies the depth prior. UBNA finds that adapting normalization statistics aggressively in shallow layers is what aligns domains in segmentation. Adaptive Normalized Representation Learning for Generalizable Face Anti-Spoofing finds that shallow layers want per-instance normalization to remove lighting and style, while deep layers need batch statistics. TTN learns a sensitivity prior showing that shallow layers are more shift-sensitive. Learned mixtures of normalizers, from Switchable Normalization, SSN and U-GAT-IT's AdaLIN, show that layers settle on different normalizers by depth. All of this work is on convolutional networks and on normalization statistics. None of it asks what the depth gradient implies for how to spend a weight-update budget in a transformer.
On the LoRA side, MoLA shows that capacity should vary with depth. It finds that putting more LoRA experts in deep layers ("inverted triangle") is best for LLaMA-2-7B. However, it leaves normalization frozen and allocates by architecture family rather than by a measured property. Normalized Low-Rank Adaptation and Norm-Bounded Low-Rank Adaptation "normalize" only the LoRA factors themselves, as an optimization and forgetting fix, and never touch the model's feature normalization layers. We reuse NoRA's initialization as a component, because it is orthogonal to our question. Parameterizations that separate weight magnitude from direction exist in the broader LoRA literature but are not in this file. A per-column magnitude on $W_0$ is close to our scale component, but it is learned per matrix rather than shared through the normalization layer. If an implementation is available, we include one as an extra baseline.
Research Program
The first stage builds the diagnostic. On ViT-B/16 we fully fine-tune on each VTAB-1k task and on FaceForensics++, recording $\Delta W$ for every projection. For each normalization layer $\ell$, we stack the matrices it feeds into a single matrix $W_0^{(\ell)}$, with update $\Delta W^{(\ell)}$. We then solve
$$\min_{s,\,B,\,A}\ \big\|\Delta W^{(\ell)} - W_0^{(\ell)}\mathrm{diag}(s) - BA\big\|_F^2, \qquad \operatorname{rank}(BA)\le r,$$
by alternating closed-form least squares in $s$ with a truncated SVD for $BA$. The scaling fraction $\rho_\ell$ is the share of $\|\Delta W^{(\ell)}\|_F^2$ explained by the scale term. We also report how $\rho_\ell$ depends on the order of fitting: fitting the scale term first versus fitting the low-rank term first. This shows whether the two parts are really separable or are simply absorbing each other's work. Three predictions follow from the depth prior. $\rho_\ell$ should fall with depth. The crossover depth should be shallower on VTAB's "natural" and "specialized" groups than on "structured", because the structured tasks (counting, depth, orientation) require new geometry. And the crossover should be shallow on face-forgery transfer, where the relevant artifacts are low-level.
Full fine-tuning is the expensive reference, so the second stage replaces it with a cheap probe. We run a few hundred steps of joint LoRA plus normalization tuning everywhere, then compute the share of update energy carried by normalization parameters in each layer. We check that this estimate ranks layers the same way as $\rho_\ell$, using Spearman correlation across tasks.
The third stage is Depth-Split Adaptation. Given the probe profile, layers whose estimated $\rho_\ell$ exceeds a threshold get normalization scale and shift tuning only. The remaining layers get LoRA, with NoRA initialization, at a rank chosen so that total trainable parameters equal a fixed budget. The threshold is the single extra hyperparameter, and it is tuned within the same trial budget as every baseline's hyperparameters.
Baselines
The naive reference points are linear probing and full fine-tuning. The latter also gives the ceiling and the update matrices used by the diagnostic. The Visual Task Adaptation Benchmark reports 65.6% mean accuracy for supervised ImageNet pretraining under its original protocol. That figure is the old reference, not a competitor. On VTAB-1k, the strongest existing method in our file is PACE. It adds multiplicative noise to adapter features and a consistency loss between two forward passes, and it reports 79.0% mean accuracy. It is the strongest competitor because it targets exactly the generalization gap we claim to close, but by a different route. CaRA is the strongest parameter-count competitor: about 2% above LoRA with 0.06M parameters versus 0.29M. No absolute VTAB number for CaRA is given in our file, so we reproduce it.
On face forgery, the reference is ViT with LoRA and Single-Center Loss. It reports 0.792 average AUC in leave-one-out cross-manipulation on FF++ and 71.69% average AUC across CelebDF, DFD and DFDC.
Pure normalization tuning, in the style of LN-tuning, is run as its own baseline. So is LoRA with an inverted-triangle depth allocation in the style of MoLA. That variant uses rank rather than experts, to keep routers out of the comparison. The ablations that isolate the key ingredient are three. Joint LoRA plus normalization tuning at every layer is the habitual recipe. A reversed split puts LoRA below the boundary and normalization tuning above it. A shuffled split uses the same per-layer assignment counts, with the measured profile permuted across depth.
All methods are run on the same backbone, data splits and epoch count, with matched trainable-parameter budgets where the method allows. Every method gets the same number of hyperparameter trials. Normalization parameters get their own learning rate, because Tuning LayerNorm in Attention documents how brittle they are to it. We report three seeds.
Beating PACE means exceeding 79.0% VTAB mean at equal or lower parameter count, using a single forward pass. Beating Kong et al. means improving cross-dataset AUC over 71.69% without their extra loss. Because Single-Center Loss is orthogonal to our method, we also report results with it added. The ablation result that matters most is the gap to joint-everywhere tuning on out-of-distribution metrics.
Why not just…?
Why not just unfreeze the normalization layers alongside LoRA everywhere? This is the real threat, because it can in principle learn the split on its own. Two pieces of evidence suggest it will not. The LN-tuning paper's finding that normalization tuning conflicts with MLP adapters suggests the two parameter sets interfere rather than divide the work cleanly. And if shallow LoRA absorbs domain artifacts, giving it the option is harmful no matter what the normalization parameters do. The joint-everywhere ablation settles this. If it matches Depth-Split Adaptation on both in-distribution and cross-dataset metrics, the contribution shrinks to the diagnostic, and we will report it as such.
Why not just tune normalization layers alone, since that already beat LoRA on multimodal LLMs? The weight-space equivalence says scale tuning cannot add directions missing from $W_0$. Tasks that need new structure, such as VTAB's structured group, should therefore suffer. The LN-only baseline's per-group VTAB scores test exactly this. The low-rank remainder in deep layers from the diagnostic is the predicted reason.
Why not just allocate more LoRA rank to deep layers, as MoLA does? An inverted-triangle LoRA allocation may capture most of the gain without involving normalization at all. The comparison that separates the two is the inverted-triangle baseline against Depth-Split Adaptation at matched parameters. If the shallow normalization tuning contributes nothing over simply dropping shallow LoRA, the normalization half of the proposal fails. We will also report a variant with shallow layers fully frozen to make this explicit.
Why not just fix LoRA's optimization with NoRA or bound its norm with NB-LoRA? These fix how a low-rank update is learned, not which kind of update each layer needs. They are orthogonal to our question, so we fold NoRA into our method and into the LoRA baselines rather than competing against it.
Decisive Checks
Three outcomes decide the proposal. First, the diagnostic must show a monotone fall in $\rho_\ell$ with depth on ViT-B. Its crossover must shift with task type in the predicted direction across VTAB's three groups and FF++. A flat profile falsifies the premise. Second, the reversed split must underperform the measured split on cross-dataset face-forgery AUC and on the VTAB natural group. If reversing the split costs nothing, depth placement is irrelevant and the method is not justified. Third, Depth-Split Adaptation must match joint-everywhere tuning in-distribution while beating it out of distribution.
As a transfer check, we repeat the diagnostic on LLaMA-2-7B fine-tuned for ScienceQA, where MoLA reports 92.36% against 91.01% for standard LoRA. We test whether the language decoder's profile reconciles the LN-tuning paper's output-heavy normalization updates with our shallow-heavy prediction. We also measure expected calibration error, following On the Calibration of Large Language Models and Alignment, to see whether moving shallow adaptation into normalization scales better preserves the pretrained model's confidence estimates.
References
- Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
- Unsupervised BatchNorm Adaptation (UBNA): A Domain Adaptation Method for Semantic Segmentation Without Using Source Domain Representations
- Adaptive Normalized Representation Learning for Generalizable Face Anti-Spoofing
- TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptation
- Differentiable Learning-to-Normalize via Switchable Normalization
- SSN: Learning Sparse Switchable Normalization via SparsestMax.
- U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
- MoLA: MoE LoRA with Layer-wise Expert Allocation
- Normalized Low-Rank Adaptation
- Norm-Bounded Low-Rank Adaptation
- The Visual Task Adaptation Benchmark
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
- Canonical Rank Adaptation: An Efficient Fine-Tuning Strategy for Vision Transformers
- Enhancing General Face Forgery Detection via Vision Transformer with Low-Rank Adaptation
- On the Calibration of Large Language Models and Alignment