Speech scientists summarize the shape of the vocal tract with a few numbers: jaw height, tongue body position, tongue tip. They usually get those numbers with principal component analysis, which finds the straight-line directions along which the shapes vary most. This works well for vowels and poorly for consonants, where the tongue presses against the palate and deforms in curved, non-additive ways. The natural fix is a kernel method, which measures similarity through a non-linear function and then runs the same linear machinery in the resulting feature space. Kernel PCA is the standard example. Kernel methods depend on a scale parameter, the bandwidth, which sets how far apart two shapes can be and still count as similar. Articulatory datasets also come from very few people, often seven to ten speakers with an unbalanced sex or dialect mix.
Our claim is that in this setting the bandwidth is not a minor tuning detail. It decides whether the non-linear coordinates describe phonetic gestures or the anatomy of whichever speakers dominate the cohort. The common default sets the bandwidth to the median distance between all pairs of shapes. That median is mostly made up of between-speaker distances, and those are mostly anatomical differences among majority-group speakers. We propose to pick the kernel scale with a held-out-speaker criterion that rewards dependence on phonetic category and penalizes dependence on speaker anatomy, with speaker groups weighted equally. We will test whether this produces consonant representations that transfer to unseen and under-represented speakers better than linear decompositions or default kernels do.
Core Claim
Bandwidth selection, cohort composition, and anatomical confounding are one problem when a non-linear articulatory map is fitted on a small cohort. If the bandwidth is too wide, the kernel's leading directions recover something close to linear PCA, and the consonantal non-linearity is lost. If it is too narrow, the Gram matrix (the table of pairwise kernel similarities) approaches the identity. Each speaker's shapes then form their own islands, and the coordinates mostly identify the speaker.
Between these extremes there is a range of scales where same-phoneme shapes from different speakers fall near one another. Where that range sits depends on who is in the cohort. We therefore expect three things. The median heuristic's bandwidth will drift as the demographic mix of the cohort is resampled. A bandwidth chosen with a speaker-held-out, phonetic-versus-anatomical criterion will be more stable. Under that criterion, kernel PCA will beat linear decompositions specifically on consonants and specifically for minority-group held-out speakers.
The new element is the selection criterion and the demographic stress test, not kernel PCA itself. If simple speaker normalization followed by linear PCA closes the gap, the claim is wrong, and the experiments are designed to show that.
Research Program
The starting point is Articulatory speaker normalisation based on MRI-data using three-way linear decomposition methods. That paper modeled midsagittal tongue contours from 7 French speakers (2 male, 5 female) across 63 articulations. It compared speaker-specific PCA, 2-Level PCA, PARAFAC and Tucker decompositions, and its authors concluded that linear models fail on consonants and that non-linear mappings should be tried. This paper motivates the work and also supplies our strongest representational baseline. An analysis of vocal tract shaping in English sibilant fricatives using real-time magnetic resonance imaging supplies the specific danger. It reports that, with few speakers, idiosyncratic palate shape and tongue asymmetry are easily mistaken for production rules. Decoupling Vocal Tract Anatomy from Speech Dynamics reports the same failure for learned models.
The method comes from the kernel literature. Kernel PCA follows the standard eigendecomposition of the centered Gram matrix, as used for appearance manifolds in Multiple View Feature Descriptors from Image Sequences via Kernel Principal Component Analysis. We use Nyström landmarks only for the larger electromagnetic articulography (EMA) corpora, following Nyström M-Hilbert-Schmidt independence criterion. That paper also measures dependence between variables with the Hilbert-Schmidt independence criterion (HSIC), a kernel measure of how much one variable tells you about another, and it explicitly leaves data-driven kernel selection open.
For a candidate bandwidth $\sigma$, we fit kernel PCA on all training speakers but one and embed the held-out speaker to get coordinates $Z_\sigma$. We then score
$$J(\sigma) = \mathrm{HSIC}(Z_\sigma, y) - \lambda\, \mathrm{HSIC}(Z_\sigma, a),$$
where $y$ is the phoneme label (delta kernel), $a$ is a vector of continuous anatomical covariates for the speaker (palate length and height, vocal tract length), and $\lambda$ trades the two terms. The score is averaged over held-out speakers with equal weight per demographic group. Two design choices keep the selector from inheriting the sensitivity it is meant to fix. First, the HSIC terms use a linear kernel on the low-dimensional $Z_\sigma$, so no second bandwidth is introduced. Second, the search over $\sigma$ is a one-dimensional grid. It is not a nested gradient loop like the one in Adaptive Kernel Selection for Stein Variational Gradient Descent, which tunes bandwidths by ascending a Stein discrepancy for particle sampling. That target is the wrong one here, because it says nothing about speaker confounding.
As an optional second step, we remove the remaining linear dependence on $a$ inside the kernel feature space. We do this with Fair Kernel Decomposition from Extending Fair Null-Space Projections for Continuous Attributes to Kernel Methods, which projects continuous protected attributes out of the Gram matrix. That paper was tested only on tabular census-style data, with a fixed, hand-tuned bandwidth. Our contribution there is to ask whether its effect survives bandwidth changes.
Kernel PCA does not give an exact way back from feature space to a tongue shape, the so-called pre-image problem. To report reconstruction error in millimeters, we learn the map from coordinates back to contour or sensor positions with kernel ridge regression on the training speakers.
Baselines
The naive reference point is the per-phoneme mean shape. It is computed on training speakers and evaluated on the held-out speaker, and it tells us how much any representation adds beyond phoneme identity.
The strongest existing representational method is 2-Level PCA from Valdés Vargas et al. That paper reports 91% variance explained and 0.11 cm RMSE with 7–11 components across 7 speakers, against 92.6% and 0.12 cm for individual 4-component PCA. These are pooled within-cohort figures, not held-out-speaker figures, and the summary we have does not split them into consonants and vowels. We therefore re-run 2-Level PCA ourselves under exactly our protocol: leave-one-speaker-out, the same component budget as kernel PCA (matched from 4 to 11), and the same reconstruction pipeline.
The strongest cheap competitor is Procrustes-aligned shapes (each speaker's contours translated, rotated and scaled to a common reference) followed by linear PCA. It removes gross anatomical scale without any kernel.
The kernel baselines are kernel PCA with the median-heuristic bandwidth and kernel PCA with bandwidth chosen by held-out reconstruction error. Both get the same grid and compute as our selector.
Two ablations isolate the key ingredient. One drops the anatomy penalty ($\lambda = 0$). The other keeps the penalty but drops equal group weighting. A further cheap ablation is the "group-balanced median": the median of pairwise distances with pairs weighted by inverse group frequency. If this matches our criterion, the simpler rule wins and we will report it.
Beating the baselines means lower held-out-speaker consonant RMSE and higher minimal-pair separability for minority-group speakers than Procrustes-plus-PCA at matched dimension. It also means equal or better results on vowels, and a selected bandwidth whose spread across resampled cohorts is smaller than the median heuristic's.
For the downstream inversion check, the reference is speaker-independent acoustic-to-articulatory inversion (predicting articulator motion from audio). Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet reports 1.08 mm average RMSE and 0.82 Pearson correlation on native speakers of EMA-MAE. Speaker-Independent Acoustic-to-Articulatory Speech Inversion reports 0.784 correlation on MOCHA-TIMIT. We do not expect a kernel representation to beat these trajectory numbers. The question is narrower: whether inverting onto our coordinates instead of raw sensors raises minimal-pair separability for held-out accented speakers without losing more than a small amount of correlation.
For Fair Kernel Decomposition on speech, no reported result was found on Lacuna. Its Nyström result, that 25–50% of components match full computation, guides only our landmark budget.
Why not just…?
Why not just normalize speakers with Procrustes alignment or vocal tract length normalization, then use linear PCA? This is the real threat. Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion warps the acoustic spectrum and gains 7% absolute correlation, yet stays about 16% worse than speaker-dependent models. Its normalization also acts on acoustics, not tract shape, so it is not a direct competitor for the representation. Procrustes alignment is a direct competitor and may well capture most of the anatomical scale effect. We therefore also run our kernel method on Procrustes-aligned shapes. If aligned linear PCA matches it on consonants for held-out minority speakers, we will conclude that the non-linearity is unnecessary once anatomy is normalized.
Why not just use a neural autoencoder with a speaker-adversarial loss? It is non-linear and avoids bandwidths. However, the French MRI corpus has about 441 shapes, and the EMA cohorts have 8–10 speakers. With so few speakers an adversary has very few distinct identities to learn from. Training Articulatory Inversion Models for Interspeaker Consistency shows that learned consistency objectives inherit corpus composition: English-trained consistency hurt Russian. We include a small autoencoder with an adversary as an extra baseline under the same leave-one-speaker-out split. A poor result would support our claim, but only for these cohort sizes.
Why not just tune the bandwidth by cross-validated reconstruction error? Reconstruction error rewards narrow bandwidths that memorize individual speakers, which is the confounded regime. This is analogous to the "metric inversion" reported in Towards Robust Evaluation of Protein Generative Models: A Systematic Analysis of Metrics, where a badly chosen MMD bandwidth ranks a corrupted model above a good one. MMD (maximum mean discrepancy) is a kernel distance between two sets of samples. This baseline is in the comparison. Our prediction is that it selects a smaller $\sigma$ with higher speaker leakage, and that is directly measurable.
Why not use a bandwidth-free alignment, such as the statistical-manifold approach? Domain Adaptation on the Statistical Manifold avoids MMD's sensitivity by using the Hellinger and Fisher-Rao geometry of distributions, reaching 93% against 88.5% for an MMD-based method on Webcam→DSLR. It aligns whole domains, however. It does not produce per-frame articulatory coordinates, so it serves as an evaluation-side comparison rather than a replacement.
Proposed Experiments
The primary data is the French midsagittal MRI corpus, if obtainable. Its documented 63 articulations separate vowels from velar, palatal and sibilant consonants. We pair it with XRMB, whose 46 speakers allow repeated subsampling of 7–10-speaker cohorts with controlled sex ratios (from balanced to 5:2 either way). EMA-MAE provides a native-versus-Mandarin-accented split, which serves as a natural demographic shift. MOCHA-TIMIT and HPRC support the downstream inversion check.
Metrics follow from the claim. The first is held-out-speaker RMSE, split into vowels and consonants. The second is minimal-pair separability in coordinate space, which McGhee et al. found tracks word error rate at −0.96 for English. The third is speaker leakage, measured both as linear HSIC with anatomical covariates and as speaker-classifier accuracy. The fourth is the spread of the selected bandwidth across resampled cohorts.
The analysis has three decisive checks.
The first is a sensitivity map. We sweep $\sigma$ over three orders of magnitude and plot phonetic dependence against anatomical dependence for each cohort composition. This shows whether a separating range exists at all and whether the median heuristic moves with the sex ratio. If the median sits stably inside the good range, our bandwidth argument fails.
The second is the consonant test. With component count matched to 2-Level PCA, the selected-bandwidth kernel PCA must beat Procrustes-plus-PCA on consonant RMSE and separability for held-out minority-group and accented speakers. If the gain appears only on vowels or only on majority speakers, the non-linearity claim fails.
The third asks whether Fair Kernel Decomposition helps after bandwidth selection or only substitutes for it. We apply it at the median, the reconstruction-optimal and the selected bandwidths. If its benefit disappears once the bandwidth is chosen well, the selection criterion is doing the real work.
References
- Articulatory speaker normalisation based on MRI-data using three-way linear decomposition methods
- An analysis of vocal tract shaping in English sibilant fricatives using real-time magnetic resonance imaging
- Towards Automatic Speech Identification from Vocal Tract Shape Dynamics in Real-time MRI
- Multiple View Feature Descriptors from Image Sequences via Kernel Principal Component Analysis
- Nyström M-Hilbert-Schmidt independence criterion
- Adaptive kernel selection for Stein Variational Gradient Descent
- Extending Fair Null-Space Projections for Continuous Attributes to Kernel Methods
- Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet
- Speaker-Independent Acoustic-to-Articulatory Speech Inversion
- Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion
- Training Articulatory Inversion Models for Interspeaker Consistency
- Towards Robust Evaluation of Protein Generative Models: A Systematic Analysis of Metrics
- Domain Adaptation on the Statistical Manifold