When a language model pretrained on clean, formal text is fine-tuned on tweets, disaster posts, or offensive social-media content, its accuracy drops. The usual explanation is representational: the model does not know the words, hashtags, or pragmatics. This note tests a second, optimization-side explanation that nobody has measured. Unfamiliar text may produce heavy-tailed gradient noise, meaning training steps that are occasionally huge rather than merely noisy. Under heavy-tailed noise, standard optimizers such as Adam behave well on average but badly in individual runs. If that is true, part of the domain-mismatch gap lives in the unlucky tail of the run-to-run distribution, and optimizers that come with high-probability guarantees (guarantees that hold for a single run with probability $1-\delta$, not only on average over imagined reruns) should recover it. They should recover more as the mismatch grows.
Put in two sentences: we measure how unfamiliar the target text is to the pretrained model and how heavy-tailed its fine-tuning gradients are, across informal-text benchmarks and controlled formal-to-informal mixtures. We then test whether the tail heaviness predicts which optimizers fail in a single run, and whether clipped or normalized optimizers, which provably tame such tails, close a mismatch-dependent part of the gap.
Core Claim
The high-probability optimization literature has established a sharp mechanism but tested it only on in-distribution or clean data. Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed proves that unclipped Adam and AdaGrad have high-probability bounds that grow like an inverse power of $1/\delta$ when noise is heavy-tailed. The same paper shows that clipped variants restore polylogarithmic dependence, and it observes empirically that CoLA fine-tuning has heavy-tailed noise while other GLUE tasks do not. It never asks why some datasets are heavy-tailed. From Gradient Clipping to Normalization for Heavy Tailed SGD shows that normalized SGD achieves $\log(1/\delta)$ dependence without knowing the tail index, but it validates only on Penn Treebank and WikiText-2. From Optimization to Generalization under Heavy-Tailed Data: The Role of Gradient Clipping proves that when per-example gradients have tail index $\alpha\in(1,2)$, the empirical variance grows with dataset size as $N^{2/\alpha-1}$. Collecting more data therefore makes unclipped SGD less stable, but the theory assumes that training and test data come from the same distribution.
On the other side, the domain-mismatch papers document the gap and fix it with data. fBERT retrains BERT with masked-word prediction on 1.4M offensive tweets. Detecting Perceived Emotions in Hurricane Disasters pretrains on unlabeled disaster tweets. I Learn Better If You Speak My Language shows that text with low perplexity under the model being trained is learned more efficiently. None of these papers looks at the optimizer, or at more than average performance.
Our claim has three parts, and each can fail on its own. First, the perplexity of the target text under the pretrained model predicts the tail index $\hat\alpha$ of per-example gradient norms at the start of fine-tuning: more unfamiliar text gives smaller $\hat\alpha$ and heavier tails. Second, smaller $\hat\alpha$ predicts a fat lower tail in the distribution of final scores across seeds for AdamW, beyond what the gradient variance alone predicts. Third, the advantage of clipped or normalized optimizers is concentrated in that lower tail and grows as mismatch grows. The link to theory is direct. The high-probability bounds in the cited work have a failure-probability dependence governed by $\alpha$. If mismatch lowers $\alpha$, then mismatch should widen the gap between the $\delta$-quantile of performance for unclipped and clipped methods.
Research Program
Measurement. For each target dataset and each pretrained model, we compute the pretrained model's mean perplexity on the target inputs. We also record out-of-vocabulary token rate, a cruder familiarity signal. We then run the first few hundred fine-tuning steps and record per-example gradient norms, estimating $\hat\alpha$ with a Hill-type estimator on the top order statistics. We check this estimate against the robustness diagnostic in Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed. Because a tail estimate from a few hundred steps is fragile, we report it with bootstrap intervals, and we also track it along training to see whether tails thin as the model adapts.
Controlled mismatch. Correlations across naturally different datasets confound mismatch with task difficulty and label noise. To separate them, we build mixtures that keep the task fixed and vary only the register of the text. For offensive-language detection we mix formal-register text with SOLID tweets at proportions from 0 to 1 and hold the label distribution fixed. For language modeling we mix WikiText-2 with SOLID text at the same proportions. We also add a label-noise control arm, following the concern raised in Can gradient clipping mitigate label noise? that clipping gains may reflect noisy labels rather than heavy tails. In that arm we inject label noise into formal data with no register change. If $\hat\alpha$ falls with the tweet proportion but not with injected label noise, mismatch rather than annotation noise is driving the tails.
Reliability evaluation. Every configuration is fine-tuned with 20 or more seeds. We report the full score distribution: mean, median, 10th percentile, and worst run. We also report the empirical $\delta$-quantile of final target loss for $\delta\in\{0.1,0.05\}$, which is the quantity the high-probability bounds actually control.
Modest theory. We extend the clipped-SGD and normalized-SGD high-probability analyses to a fine-tuning objective written as a mixture $(1-\lambda)P+\lambda Q$. Here $P$ is familiar formal text and $Q$ is informal text whose per-example gradients have a heavier tail index $\alpha_Q<\alpha_P$. The target is a bound whose $\delta$-dependence is governed by $\min(\alpha_P,\alpha_Q)$ once $\lambda$ exceeds a threshold. This would predict a qualitative change in reliability at some mixture proportion, which the mixture experiments can confirm or refute. We do not attempt target-risk transfer bounds of the kind in High Probability Risk Control Under Covariate Shift. That work controls statistical risk under shift but says nothing about optimizer trajectories, and our question is about trajectories.
Baselines
The naive reference point is standard practice: AdamW with global-norm clipping at 1.0, linear warmup, and the learning rate and schedule recommended for each model. It is run with the same seeds, data, and step budget as everything else. Its fixed clipping threshold is exactly the kind that From Gradient Clipping to Normalization for Heavy Tailed SGD argues is either inactive or always active. Beating it means a higher 10th-percentile score and a lower $\delta$-quantile loss with a mean no worse than AdamW's.
The strongest optimizer competitor is Clip-Adam-Norm from Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed, alongside normalized SGD with momentum from From Gradient Clipping to Normalization for Heavy Tailed SGD. These are the methods with the theory we rely on. The proposal does not claim to beat them. It claims that their benefit over AdamW scales with measured mismatch, and that success is a positive, significant slope of lower-tail gain against perplexity and against $1/\hat\alpha$. Every optimizer, AdamW included, gets the same tuning budget: an equal-size grid over learning rate and threshold, selected on median dev score so that no method is tuned toward its tail.
The strongest data-side competitor is domain-adaptive pretraining in the style of fBERT and HurricaneExt, run for an equal number of token updates. For this baseline the file gives reported numbers. On HatEval, fBERT reaches a Macro F1 of 0.596 against 0.483 for vanilla BERT (fBERT). On HurricaneEmo, task-guided pretraining raises BERT accuracy from 64.1% to 68.2% (Detecting Perceived Emotions in Hurricane Disasters). On TweetQA, fine-tuned BERT reaches 69.6 BLEU-1 against 78.2 for humans (TWEETQA). No reported result was found on Lacuna for clipped or normalized optimizers on any of these informal benchmarks, or for the across-seed spread of any of them. We will reproduce the vanilla and adapted numbers before comparing.
The key ablation isolates tail control from step-size control: Adam with a learning rate lowered until its median update norm matches Clip-Adam-Norm's, but with no clipping. If this matches the clipped method's lower tail, then the gain comes from smaller steps, not from cutting off rare large gradients.
Why not just…?
Why not just run several seeds and keep the best on the dev set? For BERT-size fine-tuning this is a real threat, because reruns are cheap and best-of-$k$ selection directly removes the bad lower tail. We will report best-of-$k$ AdamW for $k\in\{2,4,8\}$ at matched total compute and treat it as a genuine competitor. The proposal keeps its value in two places even if best-of-$k$ wins at small scale. The first is prediction: if perplexity measured before training forecasts tail risk, practitioners know when reruns are needed. The second is expensive single runs such as domain-adaptive pretraining, where fBERT took about five days on one V100 (fBERT) and reruns are not cheap.
Why not just do domain-adaptive pretraining and forget the optimizer? It may be that domain-adaptive pretraining works partly because it thins the gradient tails, which would make the two explanations one. We test this directly by measuring $\hat\alpha$ on the downstream task before and after adaptation. If adaptation raises $\hat\alpha$ to formal-text levels and removes the clipping advantage entirely, the optimizer story adds nothing beyond "adapt first," and we will say so. We also measure the tails during adaptation on raw SOLID text, since that stage trains on the most unfamiliar data of all.
Why not just lower the learning rate or lengthen warmup? Smaller steps also shrink the damage from rare large gradients, and Heavy-tailed Streaming Statistical Estimation shows that a delayed step schedule alone prevents early divergence. The matched-update-norm ablation above answers this. Theory predicts that at matched average step size, clipping still wins on the $\delta$-quantile because it treats rare large gradients differently from typical ones. If it does not, that prediction fails on real text.
Why not just measure gradient variance instead of a tail index? Variance is easier to estimate. But From Optimization to Generalization under Heavy-Tailed Data shows that empirical variance is an unstable, sample-size-dependent quantity when $\alpha<2$. We will fit both predictors against the lower-tail gap and report which explains more. If variance explains as much, we drop the tail index from the recommendation.
Proposed Experiments
We fine-tune BERT-base and RoBERTa-base on TweetQA, HurricaneEmo, HatEval, and OLID, with CoLA as the in-domain heavy-tail control from prior work, and run the mixture series described above. The first decisive check is whether $\hat\alpha$ falls monotonically with tweet proportion in the mixtures while staying flat under injected label noise. A flat $\hat\alpha$ refutes the central mechanism. The second is whether the lower-tail gain of Clip-Adam-Norm and normalized SGD over AdamW grows with perplexity across benchmarks and mixtures. The third reinterprets the "better model trap," in which RoBERTa underperforms BERT on disaster tweets (Detecting Perceived Emotions in Hurricane Disasters). We ask whether RoBERTa's deficit sits in the median or in the lower tail of seeds, and whether clipping closes it. If the medians match and only the tail differs, the reported gap was partly an optimization-reliability effect. If the medians differ, the representational explanation stands.
Finally, we run short domain-adaptive masked-language-model continuations on SOLID with AdamW and with normalized SGD for 5 seeds each. We record loss spikes and final downstream Macro F1 on HatEval against the reported 0.596 of fBERT, which tests whether the long single run that data-side adaptation depends on is itself a heavy-tail reliability risk.
References
- Gradient Clipping Improves AdaGrad when the Noise Is Heavy-Tailed
- From Gradient Clipping to Normalization for Heavy Tailed SGD
- From Optimization to Generalization under Heavy-Tailed Data: The Role of Gradient Clipping
- fBERT: A Neural Transformer for Identifying Offensive Content
- Detecting Perceived Emotions in Hurricane Disasters
- I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated Responses
- Can gradient clipping mitigate label noise?
- High Probability Risk Control Under Covariate Shift
- TWEETQA: A Social Media Focused Question Answering Dataset
- Heavy-tailed Streaming Statistical Estimation