A speech recognizer trained on one group of speakers, say adult men, often fails on speakers whose voices are shaped differently, such as women or children. Much of that difference comes from vocal tract length: a shorter tract shifts all resonant frequencies upward by a roughly constant factor. This note asks whether a small neural acoustic model that has already fit its training speakers perfectly keeps moving toward speaker invariance afterwards, slowly and silently. The alternative is that it never gets there unless it is shown warped voices. The question comes from grokking, or delayed generalization, where a network reaches near-perfect training accuracy long before its test accuracy rises, sometimes by thousands of epochs. We use an old, cheap tool to answer it. Linear vocal tract length normalization (VTLN) expresses any frequency warp as a small precomputed matrix applied to cepstral features. That lets us check, at every training checkpoint, how much of the model's error on unseen speakers would disappear if their vocal tracts were the right length.
The core claim is that cross-speaker generalization in data-limited neural acoustic models has a measurable delay. That delay can be split into two causes that predict different things. If the delay comes from slow, weight-decay-driven simplification of the network after it fits the data, it should scale as the inverse of the weight decay strength, and augmentation should not remove it. If it comes from the training speakers covering too narrow a range of warps, it should vanish when that range is filled in, whatever the weight decay. Telling these apart decides between two kinds of fix: optimizer-side interventions from the grokking literature, or data- and invariance-side interventions from the speaker-normalization literature.
Background and What Is Missing
Grokking theory describes training in two phases. A fast phase drives training loss to zero. A slow phase follows, in which weight decay pulls parameters along the set of zero-loss solutions toward lower norm, over a time proportional to one over the weight decay. A Theoretical Framework for Grokking derives this as a constrained norm-minimizing flow, and The Geometry of Grokking shows that the loss gradient becomes orthogonal to that solution set, leaving weight decay in control. Both analyses mostly assume squared-error loss and small weight decay, so it is open whether they describe cross-entropy acoustic models. Omnigrok supplies the experimental recipe we reuse: large initialization plus small training sets produce delays on MNIST, IMDb with LSTMs, and QM9. It never touches speech.
Two further papers suggest the delay may not be purely an optimization effect, and they motivate our competing hypothesis. Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels shows that a feature-learning kernel stays stuck when the training split respects the target's symmetry, and generalizes once that symmetry is broken. Egalitarian Gradient Descent attributes delays to badly conditioned gradients and removes them by whitening the gradient, reporting 116 versus roughly 5,392 epochs on modular addition. Vocal tract scaling is close to a continuous one-parameter symmetry of speech, approximately a shift along a log-like frequency axis. A cohort of only male speakers samples a narrow band of it. Neither line of work has asked whether that coverage gap behaves like their symmetry traps.
Linear VTLN supplies the measuring instrument. Because cepstra are a linear transform of log filterbank energies, a frequency warp $\alpha$ can be applied as a matrix, $c_{\text{warped}} = A_\alpha c$. This is derived through warped inverse cosine transforms by Panchapagesan, through bandlimited interpolation by Revisiting VTLN using linear transformation on conventional MFCC, and summarized as roughly a tenfold speedup over recomputing the front end in Linear Transformations for Vocal Tract Length Normalization. Using VTLN for broadcast news transcription shows that the linear and signal-level warp factors agree closely (correlation 0.9812). That paper also reports that VTLN's gains shrink as acoustic models get richer.
This proposal reinterprets that last finding rather than contradicting it. Richer models may eventually absorb vocal tract variation on their own, and the open question is how long that takes. Using VTLN matrices for rapid and computationally-efficient speaker adaptation and A study on combining VTLN and SAT are the closest existing combinations of linear VTLN and speaker variability. Both work only with Gaussian-mixture hidden Markov models and adaptation accuracy at convergence. Neither studies a neural model's training trajectory, which is the object studied here.
Core Measurement
For a checkpoint at training step $t$ and a held-out speaker set $S$, we define the normalization gap
$$G(t) = E_{\text{raw}}(t) - \frac{1}{|S|}\sum_{s \in S} \min_{\alpha \in \mathcal{A}} E_s(A_\alpha;\, t),$$
where $E_{\text{raw}}(t)$ is the error on the unmodified test features, $E_s(A_\alpha; t)$ is speaker $s$'s error after applying warp matrix $A_\alpha$, and $\mathcal{A}$ is a grid of roughly 20 warp factors. $G(t)$ is the part of the cross-speaker error that a per-speaker vocal tract correction would remove. A model that has become invariant to vocal tract length has $G(t) \approx 0$.
The matrix form is what makes dense tracking practical. Every checkpoint is evaluated under every warp with one batched matrix multiply on cached features, with no feature re-extraction. We do not oversell the speed, since GPU front ends are not slow. The more important property is that the transform family is exact and controlled, so $G$ isolates vocal tract mismatch from other speaker differences. A second internal measure follows LinearVC, which found speaker identity occupying a linear subspace of WavLM features of rank around 100, separate from a rank-16 phonetic subspace. At each checkpoint we fit a linear speaker-identity probe on hidden activations and record its accuracy and the rank needed to reach it.
We then define the delay $\tau = t_{\text{gen}} - t_{\text{fit}}$. Here $t_{\text{fit}}$ is when in-cohort training error crosses a threshold, and $t_{\text{gen}}$ is when $G(t)$ falls below a fixed fraction of its value at $t_{\text{fit}}$. The norm-drift hypothesis predicts $\tau \propto \lambda^{-1}$ for weight decay $\lambda$, which follows from the $1/\lambda$ slow-phase timescale in the cited theory. It also predicts that $\tau$ is largely insensitive to how widely the training warps are spread. The coverage hypothesis predicts that $\tau$ collapses as warp coverage widens and depends only weakly on $\lambda$.
Research Program
We train small LSTM and Conformer frame classifiers on MFCC inputs. MFCCs are used because the $A_\alpha$ derivations hold exactly in the cepstral domain. Training runs far past the point of fitting the training data, with an Omnigrok-style factor on the initialization scale as a controlled knob.
Warp coverage is manipulated directly. We estimate each training speaker's warp factor with the linear-VTLN maximum-likelihood procedure, cross-checked against the pitch heuristic of Faria et al.. We then build training cohorts with narrow, medium, and wide warp ranges while holding the total number of utterances fixed. We also add coverage artificially by training on $A_\alpha$-warped copies of the training features, at matched compute.
The full sweep crosses four weight decay values spanning a factor of ten, three initialization scales, three coverage widths, and augmentation on or off, with three seeds each. For every run we log $G(t)$, the speaker-probe accuracy, the parameter norm, and the alignment between updates and the weight decay direction used in The Geometry of Grokking. The last measure tests whether the slow phase actually exists under cross-entropy loss.
Baselines
The naive reference is standard training with early stopping on a held-out set of in-cohort speakers. Early stopping by definition halts near $t_{\text{fit}}$. If a delay exists, this practice discards the later cross-speaker gains, and the comparison shows how much accuracy that costs.
The strongest existing method on this exact question is per-speaker linear VTLN applied at test time, using the oracle warp from the same grid. It removes vocal tract mismatch explicitly with no training changes. On the WSJ0 split we reuse (male-only training, female test), A study on combining VTLN and SAT reports 31.8% word error rate (WER) for the baseline, 6.5% for VTLN alone, and 4.2% for VTLN followed by speaker adaptive training, all with hidden Markov models. No reported result was found on Lacuna for neural acoustic models on this split or on adult-to-child TIDIGITS, so we establish our own neural numbers.
The optimizer-side competitor is Egalitarian Gradient Descent at the rank recommended by Egalitarian Gradient Descent, alongside AdamW with tuned weight decay. Its strongest reported figure is reaching 85% MNIST test accuracy in roughly 500 steps, against 43,100 for AdamW and 1,800 for Grokfast. The representation-side competitor is speaker-aware offset subtraction from Learning Speaker Aware Offsets, which reports a WER reduction from 21.8% to 19.8% on HUB4.
All methods share the same data, architecture, and number of gradient steps, with an equal hyperparameter search budget of 16 configurations each. The ablation that isolates the key ingredient is $A_\alpha$ augmentation against random-warp augmentation of matched spectral distortion that does not follow the vocal tract scaling. If random warps shorten the delay equally, the effect is generic regularization rather than coverage of the vocal tract scaling symmetry. Beating test-time VTLN means a single trained model, with no per-speaker search, matching or beating oracle-normalized error on out-of-range speakers. Beating Egalitarian Gradient Descent means a shorter $\tau$ at equal compute.
Why not just…?
Why not just use standard augmentation such as SpecAugment and skip the dynamics? Augmentation may well work, and that is exactly the coverage hypothesis. Our experiment measures how much of the delay it removes and whether the removal depends on the augmentation following vocal tract scaling. The random-warp ablation settles the second point. An augmentation that improves the endpoint but leaves $\tau$ scaling as $1/\lambda$ would show that the slow phase is a separate effect.
Why not just use a large self-supervised model like WavLM? LinearVC shows that WavLM features still carry speaker identity in a linear subspace, so invariance is not free there. A secondary run fine-tunes a small head on frozen WavLM features with the same $G(t)$ tracking. That run tests whether pretraining simply removes the delay. If it does, we report this as the boundary of the phenomenon, not as a failure.
Why not just tune AdamW or use gradient whitening? If the norm-drift hypothesis holds, these are the right fix, and the sweep across weight decay values will show it. However, Differentiable Algebra Discovery reports that generic accelerators fail completely when the obstacle is structural. The sharpest test is a narrow-coverage cohort: if Egalitarian Gradient Descent cannot shorten $\tau$ there while $A_\alpha$ coverage can, the obstacle is the data, not the conditioning of the gradients.
Why not just early-stop and apply VTLN at test time? This baseline is a real threat for deployment, and we report it. It needs a per-speaker warp search and enough audio from each speaker to run it. It also leaves open whether the network would have learned invariance by itself, which is the scientific question here.
Proposed Experiments and Decisive Checks
We use the WSJ0 male-train, female-test split described above, and the TIDIGITS adult-to-child task, where Using VTLN matrices for rapid adaptation found linear VTLN robust where unconstrained adaptation collapsed. Metrics are frame accuracy, WER or digit error rate, $G(t)$, the delay $\tau$, and speaker-probe accuracy.
The first decisive check is whether any delay exists at all. Does $G(t)$ keep falling for many epochs after in-cohort error saturates, across seeds? A flat $G(t)$ refutes the premise and ends the norm-drift branch of the study.
The second check is the slope of $\log \tau$ against $\log \lambda$ under narrow coverage. A slope near $-1$ supports norm drift, and a slope near 0 that collapses with augmentation supports coverage. A mixture of the two would show that each cause accounts for part of the delay.
The third check is whether speaker-probe accuracy in hidden layers falls during the same window that $G(t)$ falls. That would link the late invariance to the network discarding speaker information internally, the process that offset subtraction imposes by hand.
References
- A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisation
- The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold
- Omnigrok: Grokking Beyond Algorithmic Data
- Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels
- Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
- Frequency warping by linear transformation of standard MFCC
- Revisiting VTLN using linear transformation on conventional MFCC
- Implementing frequency-warping and VTLN through linear transformation of conventional MFCC
- Using VTLN for broadcast news transcription
- Using VTLN matrices for rapid and computationally-efficient speaker adaptation with robustness to first-pass transcription errors
- A study on combining VTLN and SAT to improve the performance of automatic speech recognition
- LinearVC: Linear Transformations of Self-Supervised Features Through the Lens of Voice Conversion
- Efficient pitch-based estimation of VTLN warp factors
- Learning Speaker Aware Offsets for Speaker Adaptation of Neural Networks
- Differentiable Algebra Discovery Accelerates Grokking via a Non-Fourier Mechanism