Do Self-Supervised Speech Models Hear Formants, or Only Vowels?

Formants are the resonant frequencies of the vocal tract. They are the standard way phoneticians, clinicians and sociolinguists measure how a vowel was actually produced. They are also hard to measure well. Classical peak-picking tools disagree with each other on children's speech, and small neural trackers trained on adult recordings break down when the speaker's vocal tract is shorter or the pitch is higher. Self-supervised speech models such as wav2vec 2.0 learn from thousands of hours of unlabeled audio and are known to carry articulatory information in their middle layers. That makes them a tempting source of features for a tracker that works across ages and accents without calibration. The catch is that nobody knows whether these models represent continuous resonances at all. They might represent only the vowel category, which a probe could then map to a typical formant value. On adult benchmarks such a probe would look accurate while measuring nothing.

This note proposes a formant tracker built on frozen self-supervised features, paired with a test of whether its outputs follow the acoustics. The test resynthesizes speech with controlled formant shifts while holding vowel identity fixed, then checks whether the tracker follows the shift. The main claim has three parts: intermediate self-supervised layers encode resonance frequencies beyond vowel identity; this information transfers to children's speech better than a spectrogram-trained tracker does; and the resynthesis test can tell real formant reading from category lookup, which ordinary error metrics cannot.

What exists and what is missing

Two lines of work almost meet here. Probing studies show that wav2vec 2.0 layers carry phonetic structure. Domain-Informed Probing of wav2vec 2.0 Embeddings for Phonetic Features classifies phoneme, manner and place from phoneme-averaged TIMIT embeddings and finds a sharp rise around layer 7. Using wav2vec 2.0 for phonetic classification tasks: methodological aspects shows that nasality probing improves from 81.04% to 94.05% with ±1 s of context, checked against airflow recordings. Exploring Self-Supervised Speech Representations for Cross-lingual Acoustic-to-Articulatory Inversion regresses tongue and lip sensor positions from wav2vec 2.0 and MMS layers. It finds that layers about two-thirds of the way up work best, that ASR fine-tuning hurts, and that MMS-1b reaches a cross-lingual Dutch correlation of 0.564 against 0.330 for MFCCs. These papers motivate the proposal, but none targets continuous acoustic resonances, and the first two discard within-phoneme dynamics.

The second line is supervised formant tracking from spectral input. Examples include the dilated temporal convolutional network of Formant Tracking Using Dilated Convolutional Networks Through Dense Connection with Gating Mechanism and the heatmap classifier of Formant Estimation and Tracking using Probabilistic Heat-Maps. These supply the output head and the training data, the hand-corrected Vocal Tract Resonance (VTR) trajectories. They have never been given pretrained self-supervised input.

Vowel Normalisation in Latent Space for Sociolinguistics acts as a constraint. In a small autoencoder trained on vowel spectra, formants were spread over 4–5 entangled dimensions. Decodability alone therefore says little about how the model stores them. Audio Transformer for Synthetic Speech Detection via Multi-Formant Analysis used Praat formant tracks as an auxiliary target to keep a deepfake detector honest. We do the reverse: we ask whether a general pretrained model already carries that signal.

Speaker-independent neural formant synthesis maps F0, F1–F4, spectral tilt and energy to speech through HiFi-GAN. We reuse it as the source of the counterfactual stimuli described below. The paper notes that accuracy degrades at large scaling factors, which limits us to moderate shifts.

Research Program

We extract frame-level features from every layer of frozen wav2vec 2.0 base and large, MMS-300m and MMS-1b, and from a wav2vec 2.0 model fine-tuned for ASR. The convolutional encoder output is included as layer 0. These features have 20 ms receptive frames at a 10 ms stride, which matches VTR's frame rate. On each layer we train two probes on VTR to predict F1, F2 and F3 per frame. The first is linear. The second is the heatmap head of the probabilistic heat-maps paper, which predicts a distribution over frequency bins with F2 conditioned on F1. We also train a learned weighted sum over layers.

The probe is trained on adult VTR only, so any success on children's speech is transfer, not calibration.

The resynthesis test works as follows. We take held-out VCTK vowels, and later Hillenbrand vowels. We resynthesize each one with the neural formant synthesizer, shifting F1 or F2 by a known amount (we plan ±5% to ±20%) while copying every other parameter. The test then measures two quantities. The first is the sensitivity slope: the change in predicted formant divided by the imposed change. A slope near 1 means the tracker reads resonance. A slope near 0 means it reads category. The second is the within-vowel residual: on natural speech, how much of the deviation of each token from its vowel's mean formant the tracker explains.

Synthesis is imperfect. We therefore take the realized shift to be the shift on which the heatmap tracker and Praat agree, and we discard tokens where they disagree by more than the shift itself. A layer-wise map of all three measures (absolute error, slope and within-vowel residual) then shows where continuous resonance information lives. It also shows whether that location differs from the layer-7 phonetic peak and from the two-thirds-depth articulatory peak reported earlier.

Baselines

The naive reference point is a category predictor. It outputs the training-set mean formant for the frame's vowel label and the speaker's sex, using gold labels. It needs no audio modeling, but it will score respectably on adult VTR because formants cluster by vowel. Any self-supervised probe must beat it on absolute error. More importantly, it has a within-vowel residual and sensitivity slope of zero by construction, which makes it the reference for those two measures.

The strongest existing method is the heatmap tracker of Formant Estimation and Tracking using Probabilistic Heat-Maps. Trained on VTR and tested on Hillenbrand, it reports an F1 error of 26 Hz, against 36 Hz for DeepFormants and 68 Hz for WaveSurfer. It is the strongest competitor because it was built for exactly this cross-domain transfer. On in-domain VTR, the dilated convolutional tracker reports 8.2% mean absolute percent error and 84 Hz mean absolute error. We retrain both on the same VTR split, with the same tuning budget as our probes (an equal number of hyperparameter trials) and the same frame rate. We then report Hillenbrand results separately for men, women and children. The file gives no reported children-only number on Lacuna for the heatmap tracker, so our retrained version supplies that figure.

Beating it means two things: lower F1 and F2 error on the Hillenbrand children subset, with no loss on adults, and a sensitivity slope at least as close to 1.

Praat LPC tracking at default and age-adjusted ceilings is the zero-training reference. My Vowels Matter: Formant Automation Tools for Diverse Child Speech reports wide disagreement among such tools on child speech but no global accuracy figure.

Three ablations isolate the ingredient that matters. First, we run the identical heatmap head on log-mel and linear spectrograms, which separates pretraining from head design. Second, we run it on a randomly initialized wav2vec 2.0, which separates learned features from architecture. Third, we run it on the ASR-fine-tuned model. Based on the articulatory inversion result, we predict this model will show a lower sensitivity slope while keeping similar category accuracy.

Why not just…?

Why not just use Praat or another LPC tool? These tools need no training and remain the field standard. For adult read vowels they may be adequate. Their failures concentrate where we evaluate: high-pitch child voices, low-F1 and extreme-F2 vowels, and merged peaks. There the file's evidence shows tool disagreement on children and a 68 Hz F1 error for WaveSurfer cross-domain. We report Praat on every subset, so if it matches the probes on children, the proposal's practical case collapses and only the analysis contribution remains.

Why not just train a small supervised tracker on spectrograms? This is the real threat. The heatmap tracker already transfers reasonably to Hillenbrand, and the same-head spectrogram ablation tests this alternative directly. Self-supervised models pretrained mainly on adult audiobooks may inherit the same adult bias. Comparing MMS against LibriSpeech-only models partly tests whether more varied pretraining helps. If the spectrogram head matches the best self-supervised probe on children's error and on slope, we will conclude that pretraining adds no resonance information and report that as a negative finding.

Why not just fine-tune the whole self-supervised model on VTR? VTR has only 538 utterances, so full fine-tuning risks overfitting to adult ranges. It would also blur the question of what the pretrained representation contains. We run it as a secondary condition. If it wins on adults but loses on children, the frozen design is justified. If it wins everywhere, it becomes the recommended tracker, and the resynthesis test still says whether it reads resonances.

Why not just report absolute error on VTR and skip the resynthesis test? A category lookup scores well on adult benchmarks because vowel identity predicts much of formant variance. Error alone cannot distinguish measurement from guessing. The category baseline shows how much of a probe's apparent accuracy this effect accounts for.

Proposed Experiments

The first experiment is the layer-wise map on VTR. It covers mean absolute error and percent error for F1–F3, the within-vowel residual correlation, and sensitivity slopes on VCTK resynthesis, for all models and both probe types. We expect continuous resonance information to peak at lower-intermediate layers, below the categorical peak. We also expect the ASR-fine-tuned model to keep low absolute error while its slope drops. That pattern would show it has shifted toward category lookup.

The second experiment is zero-shot transfer to Hillenbrand men, women and children, and to Clopper's female speakers, against the retrained heatmap and dilated-convolution trackers and Praat. Errors are broken down by vowel so that the low-F1 and extreme-F2 cases flagged for child speech are visible.

The third experiment repeats the resynthesis test on Hillenbrand child vowels. This is the setting where a category reader would fail, because children's formants sit far above the adult means that the probe learned.

Two outcomes would settle the question. The first is a self-supervised probe that beats the retrained heatmap tracker on children's F1 and F2 error while keeping a sensitivity slope above about 0.8. The second is a probe whose low adult error coincides with a slope near zero. That would show that these models' formant decodability is mostly vowel identity, a caution for anyone using them in clinical or sociophonetic measurement.