When a large language model is fine-tuned on instruction data, it quickly becomes nearly certain about most training tokens. At that point the standard training loss gives almost no signal on those tokens, a problem called softmax saturation. The model also grows overconfident: after instruction tuning on Alpaca, calibration error on MMLU jumps from near zero to over 0.20, even when training uses low-rank adaptation (On the Calibration of Large Language Models and Alignment). Low-rank adaptation (LoRA) is a way to fine-tune by learning a small update $\Delta W = BA$ of rank $r$ next to each frozen weight matrix. Our hypothesis is that under LoRA these two failures share one cause that matters more than it would under full fine-tuning. With only $r$ directions to spend, the adapter puts a large share of its update into sharpening, meaning it scales up the output logits along directions the model already ranks correctly. Sharpening lowers the loss on easy tokens. It does not change which answer the model prefers, and it takes rank away from the harder tokens the fine-tuning is meant to teach.
We propose to test this hypothesis directly, and then to remove the sharpening component from every LoRA update. At each step the adapter update is projected so that it is orthogonal to the gradient of the batch's mean output entropy. A single trainable temperature scalar is added to take over whatever sharpening the task actually needs. The adapter is then pushed to change rankings instead of confidence, and confidence is moved into one parameter that is easy to inspect and re-calibrate. If the hypothesis holds, we expect better calibration than vanilla LoRA and better accuracy per unit of rank. The accuracy gain should be largest at small rank and fade as rank grows. That rank-dependence is the signature that separates this idea from a calibration trick.
Core Claim
Our claim is that during supervised fine-tuning, a measurable and growing fraction of the LoRA update lies along the "sharpen what is already right" direction. This fraction is larger at low rank. Removing it during training, rather than after, improves accuracy at fixed rank and leaves the model better calibrated before any post-hoc correction.
Two prior results make the accuracy part plausible rather than only a calibration story. First, On the Convergence Rate of LoRA Gradient Descent shows that LoRA training slows down as the adapter factors move away from the origin. Sharpening inflates adapter norm, so it should also slow down all later learning. Second, Universal One-third Time Scaling in Learning Peaked Distributions shows that softmax cross-entropy on peaked targets forces logits to keep growing and caps convergence at roughly loss $\sim \tau^{-1/3}$. Logit growth is therefore not a side effect but where the optimizer is heading on easy tokens.
Relation to Prior Work
Several papers motivate the gap without filling it.
The calibration-across-alignment study (Xu et al.) measures the confidence surge and treats LoRA as an empirical safety rail. It never asks how the factorization interacts with the vanishing gradient $p - y$.
The softmax-bottleneck analysis (Why do small language models underperform?) shows that a rank-constrained output head saturates during pretraining. It studies only the head, not internal adapters during fine-tuning.
Two pretraining papers show that growth in Query–Key singular values pushes the attention softmax into one-hot collapse, and they constrain that growth with orthonormal weights or a reactive normalizer: Different Layers, Different Manifolds and One QK Channel, Many Sources. Both study full-parameter pretraining. Neither asks where sharpening goes when only a low-rank adapter can move.
Other papers supply competing or borrowed methods.
Bayesian Low-rank Adaptation for Large Language Models fixes LoRA overconfidence after training with a Laplace approximation over adapter weights. It cannot reclaim rank that was spent on sharpening during training.
LoRA-Pro aligns the low-rank update with the full fine-tuning gradient. When that full gradient is itself mostly sharpening, LoRA-Pro faithfully copies the sharpening. Our projection is complementary and can be stacked on it.
Two papers supply the methodological idea of acting during training rather than after it:
- Noisy Softmax delays saturation in convolutional networks by subtracting annealed noise from the target logit.
- Decoupled Entropy Minimization separates the confidence-driving part of an objective from the part that drives learning.
We borrow the idea of separating "become more confident" from "learn something new." We implement it as a geometric projection on the adapter update, not a change of loss.
Research Program
The first stage is diagnosis. We fine-tune Llama-3.1-8B-Base with vanilla LoRA on Alpaca and on GSM8K training data, at ranks 4, 8, 32 and 128. At each checkpoint we measure three quantities.
The first is the sharpening fraction, defined as follows. Let $g$ be the stacked LoRA gradient for a batch, and let $u = \nabla_{A,B}\,\bar H$ be the gradient of the batch-mean output entropy $\bar H$ with respect to the same adapter parameters. The sharpening fraction is $\cos^2(g, u)$. We report it separately for tokens already predicted with probability above 0.9 and for the remaining tokens.
The second is how much of each adapter's top singular directions, when removed, reduces logit norm without changing argmax predictions.
The third is the maximum attention logit in adapted Query–Key layers. This tells us whether sharpening is routed through attention, as the QK-collapse work would predict, or through the output-side projections.
The diagnosis stage has its own success condition. The sharpening fraction must rise over training and must be higher at low rank. If it is flat or rank-independent, the method below has no reason to work, and we stop.
The second stage is the method. Write the output logits as $z = \lambda\, W_{\text{head}} h$, where $h$ is the final hidden state, $W_{\text{head}}$ the frozen output matrix, and $\lambda$ a trainable scalar initialized to 1. The adapter update is replaced by its confidence-neutral projection:
$$g_{\perp} = g - \frac{\max(0,\,-\langle g, u\rangle)}{\|u\|^2}\, u .$$
Only components that reduce entropy are removed. An update that makes the model less confident is left untouched. The scalar $\lambda$ receives its ordinary gradient, so any sharpening the data truly calls for goes there. The extra cost is one additional backward pass for $u$, which we will cost against LoRA-Pro's reported overhead of roughly 10 minutes on a 90-minute run plus about 4GB of extra memory.
Two variants test where sharpening lives. The first restricts the projection to Query–Key adapters. The second restricts it to all other adapters.
Baselines
The naive reference point is vanilla LoRA run on the same data, ranks and token budget. Because A Unified Study of LoRA Variants shows that well-tuned vanilla LoRA matches or beats most variants and is very sensitive to learning rate, every method gets the same learning-rate sweep, at least five values bracketing 5e-4, with the best value chosen on held-out data. That study reports 75.51% GSM8K accuracy for vanilla LoRA on Llama-3.1-8B-Base at learning rate 5e-4. We treat that as the accuracy bar our reproduction must reach before any comparison counts. Beating it means a gain at rank 8 that exceeds the seed-to-seed spread over three seeds.
The strongest accuracy-side competitor is LoRA-Pro, which corrects the low-rank gradient toward the full fine-tuning gradient. Its paper reports GSM8K 57.57% versus 42.08% for LoRA at rank 8, HumanEval pass@1 22.97% versus 14.76%, and a GLUE average of 88.44 versus 82.08. The setup differs from the unified study, so we rerun LoRA-Pro in our own pipeline rather than compare against the quoted numbers. We also run LoRA-Pro with our projection applied after its correction. If the combination beats LoRA-Pro alone, the two fixes address different problems.
The strongest calibration-side competitor is Laplace-LoRA, which places a Kronecker-factored Laplace posterior over the adapter weights after training. It reports Winogrande-small expected calibration error falling from about 31.2% for the standard point estimate to about 2.1%. We run it on our adapters and on the common-sense suite (Winogrande, ARC, OpenBookQA, BoolQ). Beating it on calibration alone is not our goal. Matching it within one percentage point of calibration error without any post-hoc step, while also gaining accuracy, would count as a win. Laplace-LoRA applied on top of our adapters is also reported.
Three ablations isolate the key ingredient.
- The first keeps the trainable temperature but drops the projection. It tests whether simply giving sharpening a cheap outlet is enough.
- The second keeps the projection but freezes $\lambda$.
- The third projects out a random direction of matched norm. This mirrors the random-direction control in the QK-collapse paper, where removing leading spectral directions held the top singular value to 11.1 while removing random directions let it reach 237.
Why not just…?
Why not just use post-hoc temperature scaling? Temperature scaling repairs calibration after training. Parametric ρ-Norm Scaling Calibration reports it bringing CIFAR-100 ResNet18 calibration error from 0.160 to 0.033. It cannot return rank that was spent on sharpening during training, so it predicts no accuracy change. We report every method both with and without post-hoc temperature scaling. If our accuracy gain disappears and only the calibration gain remains, then temperature scaling is the cheaper answer and we will say so.
Why not just use label smoothing or a logit-norm penalty? This is the most serious threat, because both stop target probabilities from approaching one. Unlike our projection, both change the training target or loss for every token. They push predictions toward uniform even on tokens the model has not yet learned. Our projection removes only entropy-reducing components of the update. We run both, tuned over the same budget. If either matches the projection's rank-8 accuracy and calibration, the projection adds nothing and the contribution reduces to the diagnosis.
Why not just raise the rank? If sharpening is crowding out task learning at low rank, more rank might fix it just as cheaply. Our hypothesis predicts exactly that the gap between vanilla LoRA and confidence-neutral LoRA closes as rank grows. The rank sweep from 4 to 128 is therefore the test of the mechanism, not a nuisance. The method is worth adopting only if a rank-8 confidence-neutral adapter matches or beats a vanilla adapter of roughly twice the rank, at the cost of one extra backward pass.
Proposed Experiments
The main runs fine-tune Llama-3.1-8B-Base on three settings:
- Alpaca, evaluated for MMLU accuracy and calibration error, where the 0.20 surge was reported.
- GSM8K, evaluated for accuracy.
- The common-sense suite, evaluated for accuracy, negative log-likelihood and calibration error.
For each setting we cover ranks 4 to 128, the baselines and ablations above, three seeds, and matched learning-rate sweeps. A smaller Pythia sweep makes dense checkpointing of the sharpening fraction affordable. It also tests whether the loss-versus-time exponent, fitted between 0.31 and 0.37 for Pythia pretraining in the one-third-scaling paper, changes under confidence-neutral updates.
Three checks decide the outcome.
- The diagnosis must show a rising, rank-dependent sharpening fraction.
- Confidence-neutral LoRA must beat tuned vanilla LoRA, label smoothing and temperature-only training on accuracy at rank 8, with that margin shrinking at rank 128.
- Its calibration error before any post-hoc correction must come within a point of Laplace-LoRA.
If the first check fails, the hypothesis is wrong. If the second fails while the third succeeds, the result is a training-time calibration method and should be presented only as that.
References
- On the Calibration of Large Language Models and Alignment
- On the Convergence Rate of LoRA Gradient Descent
- Universal One-third Time Scaling in Learning Peaked Distributions
- Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
- Different Layers, Different Manifolds: Module-Wise Weight-Space Geometry in Transformer Optimization
- One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
- Bayesian Low-rank Adaptation for Large Language Models
- LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
- Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
- Decoupled Entropy Minimization
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- Parametric ρ-Norm Scaling Calibration