Deep networks have critical learning periods. These are short windows early in training when a defect in the data, such as blurred images, does permanent damage that later clean training cannot undo. A separate body of work shows that the flatness of the final solution, meaning how little the loss rises when the weights are nudged, is decided late. One study found that switching to a flatness-seeking optimizer for only the last few epochs gives almost all of its benefit. These two findings pull in opposite directions, and nobody has checked whether they describe the same thing.
This note proposes a direct test. We damage networks during their critical period, repair them with the strongest known late-stage flattening methods, and then measure what those methods leave behind. The measuring tool is compression. Rounding weights to 2–4 bits is a perturbation of the weights, and it reveals curvature that ordinary full-precision test accuracy hides. The concrete question is this: after late flattening, does an early deficit still show up as extra sensitivity to quantization in models that look equally accurate at full precision? If so, how flat a network can become is settled early, and schedules for compressible models should spend their curvature-shaping budget there.
Core Claim
Our hypothesis is that late-stage flattening only chooses among basins that the early transient has already made reachable. A deficit during the critical window therefore raises a floor on the curvature the network can reach at the end of training.
To make "curvature" and "quantization sensitivity" comparable, we use the standard second-order picture. Rounding weight $w_i$ to a grid with step $s_i$ adds roughly uniform noise of variance $s_i^2/12$. Near a minimum, the expected increase in loss at bitwidth $b$ is then approximately
$$\mathbb{E}[G_b] \approx \frac{1}{24}\sum_i H_{ii}\, s_i^2 ,$$
where $G_b = L(Q_b(w)) - L(w)$ is the quantization gap and $H_{ii}$ are the diagonal entries of the loss Hessian. This comes from a second-order Taylor expansion of the loss with zero-mean rounding noise. The quantization gap is therefore a cheap, step-size-weighted readout of curvature. It can differ sharply between two models with identical full-precision accuracy.
We define the scar of a deficit as the excess $G_b$ of a model that had the deficit and then received late repair, relative to a clean model given the same repair and matched in full-precision accuracy. The claim predicts three things. First, the scar is positive at low bitwidths. Second, it depends on when the deficit started, peaking at the same epochs where Critical Learning Periods in Deep Networks found maximal vulnerability to blur. Third, it is not removed by late sharpness-aware minimization (SAM, which minimizes the worst-case loss within a small ball around the weights) or by weight averaging.
Position Relative to Prior Work
Three lines of work motivate the question.
- Early training matters most. Critical Learning Periods in Deep Networks tracks the trace of the Fisher information and shows that blur over the first 40–60 CIFAR-10 epochs raises final error from about 6% to over 18%. Time Matters in Regularizing Deep Networks shows that weight decay applied only late is nearly useless, and that generalization tracks the early Fisher peak. Both measure damage only as test error.
- Flatness is decided late. Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training shows that SAM for the final 5 of 200 epochs nearly matches full SAM, and that early SAM followed by ordinary SGD is washed out. It never introduces early deficits and never measures compressibility.
- Quantization brittleness appears during the final decay. Training Dynamics Impact Post-Training Quantization Robustness finds that brittleness surges during the final learning-rate decay and that weight averaging heals it. It does not ask whether early conditions predispose a run to that late sharpening.
The closest bridge is Better Schedules for Low Precision Training of Deep Neural Networks. It frames aggressive early low precision as a critical-period violation that full precision cannot later fix. It measures no geometry and applies no late flattening. Exploring Quantization for Efficient Pre-Training of Transformer Language Models reports that 4-bit training from step zero lands in sharper minima, but it does not tie this to a time window.
Our contribution differs from each of these. We combine controlled early deficits with the late repairs that the flatness literature considers sufficient, and we score the outcome by geometry and quantization gap at matched full-precision accuracy. Two papers supply methods we reuse: Train Flat, Then Compress (SAM as a compressibility tool) and A Scale Invariant Flatness Measure for Deep Network Minima (a flatness measure that weight rescaling cannot game).
Research Program
Vision arm. This is the core. We reproduce the CIFAR-10 blur protocol of Achille et al. with a ResNet-18 and the WideResNet-16-8 used by Zhou et al.
- Deficit design. We use sliding deficit windows of fixed length (20 epochs) whose onset sweeps from epoch 0 to epoch 120.
- Late repair. After the deficit, each run gets one of four endings: plain SGD, SAM for the last 5 epochs as in Zhou et al., SAM for the last 40 epochs, or averaged weights over the final epochs.
- Measurements at convergence. We record the Hessian top eigenvalue, Hessian trace estimated with Hutchinson probes, the scale-invariant flatness measure, the SAM perturbation gap at several radii, and post-training quantization gaps.
- Quantization settings. We use uniform per-channel round-to-nearest quantization at 8, 4, 3 and 2 bits, with no calibration fine-tuning.
- Early measurements. Throughout the first 60 epochs we log the Fisher trace. This tests whether its peak predicts the final scar, the geometric analogue of the correlation reported by Golatkar et al.
Language-model arm. This tests whether the effect survives in the setting where compression matters. We train 70M-parameter transformers on FineWebEdu (HuggingFaceFW/fineweb) with a warmup-stable-decay schedule, following the controlled setup of Catalan-Tatjer et al.
- Early deficit. We use one of two deficits in the first few percent of tokens: 4-bit simulated weight quantization (the Wolfe–Kyrillidis impairment), or a low-quality token mix in the spirit of LLMs Can Get "Brain Rot"!.
- Late repair. Every run gets the healing ingredients that paper identified: weight averaging and, separately, a higher peak learning rate.
- Outcome. We measure 3- and 4-bit post-training quantization loss.
Constructive test. If scars exist, a cheap schedule should follow: SAM only during the high-Fisher window, followed by the standard late SAM switch. Zhou et al. show early SAM alone is erased by later SGD. They never paired early and late SAM, and they never measured quantization. This schedule costs a fraction of full SAM's doubled compute.
Baselines
Do nothing: clean data with plain SGD. This fixes the reference level of curvature and quantization gap. Separately, deficit runs with no repair show the size of the raw damage, which on CIFAR-10 should reproduce the ~6% to >18% error jump of Achille et al.
Strongest existing late repair. The strongest competitor is the SGD-to-SAM switch for the final 5 epochs from Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training. It reports about 3.4% test error on CIFAR-10 WRN-16-8, against about 3.2% for full SAM and 4.4% for SGD. It is the strongest competitor because it is exactly the claim that late flattening suffices, and it is cheap. We add full-run SAM, and late weight averaging as in Training Dynamics Impact Post-Training Quantization Robustness.
Fairness controls. All methods share data, architecture, epoch count and augmentation. SAM's radius and the averaging window each get the same grid-search budget, tuned on clean runs and then frozen. That way the deficit runs cannot be rescued by retuning.
Missing reference numbers. We found no reported result on Lacuna for post-training quantization gaps of CIFAR-10 models, with or without late SAM. Those numbers will be produced here.
What counts as beating late repair. The early-plus-late SAM schedule beats late SAM if two conditions hold together. First, it reduces the 3-bit and 2-bit quantization gap of deficit runs by more than seed-to-seed variation over five seeds. Second, its full-precision accuracy is no worse.
Key ablation. Early-only SAM followed by plain SGD tells us whether the early intervention works by itself or only by making late flattening effective.
Why not just…?
Why not just compare full-precision accuracy? Two models can have the same full-precision accuracy and very different quantization gaps. Q-BERT shows direct 2-bit SQuAD quantization collapsing to about 10 F1, while careful allocation reaches about 80 F1. Catalan-Tatjer et al. find robustness differences at similar validation loss. We match runs on full-precision accuracy by selecting checkpoints. If the scar vanishes once accuracy is matched, the claim fails and we will report that.
Why not just rely on late SAM or weight averaging? This is the real threat. It is also the hypothesis under test, not something to argue around. If late SAM with 40 epochs, or weight averaging, drives deficit runs to the clean runs' Hessian trace and quantization gap at every onset, then the late-flatness view wins. The practical conclusion would then be to ignore the critical window for compression.
Why not just raise the peak learning rate? Catalan-Tatjer et al. show higher peak rates improve 3- and 4-bit robustness. Achille et al., however, report that critical-period deficits persist across learning-rate schedules and optimizers. We include a high peak rate as a repair condition in both arms. A scar that survives it is evidence that the effect is not a learning-rate artifact.
Why not just use scale-invariant measures, or blame outliers? Quantization gaps can reflect weight norms or activation outliers rather than curvature. Chitsaz et al. find early-emerging outliers under 4-bit training. We report the scale-invariant flatness measure next to the raw Hessian and use per-channel quantization. We also check whether the gap tracks the Hessian-weighted prediction in the equation above. If it tracks outlier magnitude instead, we will say the mechanism is not curvature.
Decisive Checks
The onset curve. Plot the scar against deficit onset under late SAM. A peak near the Achille vulnerability epoch, with a near-zero scar for onsets after epoch ~60, supports the claim. A flat curve refutes it.
Fisher prediction. The early Fisher peak should rank-correlate with the final 2-bit quantization gap across runs, and should do so better than final full-precision accuracy does.
Language-model transfer. On the language-model arm, early 4-bit or low-quality exposure should leave a 3-bit quantization gap that weight averaging does not close. Gap-closing would mean the scar is a small-vision phenomenon.
The cheap schedule. If the early-plus-late SAM schedule closes scars that late SAM leaves, the result is a practical, cheaper-than-full-SAM recipe for compressible models. If the onset curve is flat, it cleanly supports the late-flatness view against the critical-period view.
References
- Critical Learning Periods in Deep Networks
- Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence.
- Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late In Training
- Training Dynamics Impact Post-Training Quantization Robustness
- Better schedules for low precision training of deep neural networks
- Exploring Quantization for Efficient Pre-Training of Transformer Language Models
- Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models
- A Scale Invariant Flatness Measure for Deep Network Minima
- LLMs Can Get "Brain Rot"!
- Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT