Keeping Anatomy at Home: Federated Speech Inversion with Private, Static Vocal-Tract Adapters

Recordings of how the tongue, lips and jaw move during speech are rare, expensive to collect, and increasingly gathered in clinics from children and patients. They are also fingerprints. Tongue shape alone identifies a speaker among 40 with an equal error rate of 5.6% (Articulatory Strategy in Vowel Production as a Basis for Speaker Discrimination). Federated learning is the standard response to data that cannot leave its site: each site trains locally and sends only model updates to a server. Applying it naively to articulatory data misses where the speaker-specific information lives. In articulatory speech, much of what separates one person from another is static anatomy: palate shape, jaw length, tongue size. The part worth sharing across sites is the dynamic motor pattern, meaning how constrictions are timed and blended to produce phonemes.

We propose that each site keep a small, deliberately static model of its speakers' anatomy on premises, and share only a temporal model of articulatory dynamics expressed in a common "canonical" space. The task is acoustic-to-articulatory inversion: predicting vocal-tract movements from audio. The claim is that splitting the model along the anatomy/dynamics line does two things together. It recovers most of the accuracy of centrally pooled training. It also makes the shared model measurably less identifying than either centrally trained or naively federated models. Both halves of that claim are testable on existing corpora.

Core Claim

Write $a_{1:T}$ for an utterance's acoustic features and $z_t$ for its measured articulatory trajectory in the nine tract variables (constriction degree and location at lips, tongue tip and tongue body) used by Multi-Corpus Acoustic-to-Articulatory Speech Inversion. Client $k$ predicts

$$\hat{z}_t = g_k\big(f_\theta(a_{1:T})_t\big),$$

where $f_\theta$ is a shared sequence model trained by federated averaging. It sees the whole utterance and outputs canonical trajectories. $g_k$ is a private map that never leaves the client. The key design choice is that $g_k$ is frame-wise and low-capacity: a per-channel monotone nonlinearity followed by a small cross-channel linear mix, with no temporal context. Such a map can rescale, shift and warp the space a speaker's articulators move in, which is what anatomy does. It cannot encode timing, coarticulation or phoneme identity, so those must live in the shared model.

Two features distinguish this from the two federated speech methods in the file. CAFT (Federated Acoustic Modeling For Automatic Speech Recognition) learns a per-client affine transform on the acoustic input, and it uses that transform to canonicalize recording conditions. We place the private map on the articulatory output, because that is where vocal-tract geometry enters. FedSpeech (FedSpeech: Federated Text-to-Speech with Continual Learning) isolates per-speaker sub-networks with pruning masks and a private speaker embedding. It does this to preserve voice identity, and it places no structural limit on what the private part may encode. Our private part is constrained in what it can represent. That constraint is what lets us make, and test, a privacy claim about the shared part.

The design is grounded in two lines of evidence. The first is that models trained on vocal-tract data confuse static morphology with motor dynamics (Decoupling Vocal Tract Anatomy from Speech Dynamics; Towards Automatic Speech Identification from Vocal Tract Shape Dynamics in Real-time MRI). The second is that size-related tongue shape carries strong identity information (Articulatory Strategy in Vowel Production as a Basis for Speaker Discrimination). There is also a contrary hint we must face. In that same study, shape-only features still reached 7.5% equal error rate, so some identity is dynamic and will not be removed by a static adapter. We therefore claim reduced identifiability of the shared model, not anonymity.

Research Program

Clients and data. We simulate a federation in which each speaker is a client and speakers are grouped into sites by corpus. The corpora are XRMB (56 speakers), EMA-IEEE (8), MOCHA-TIMIT and HPRC (8 each, as used by Speaker-Independent Acoustic-to-Articulatory Speech Inversion). Each corpus is mapped to the shared tract variables following Multi-Corpus Acoustic-to-Articulatory Speech Inversion. This gives within-site anatomical variation and across-site sensor variation, which is the realistic non-IID structure.

Acoustic input comes from a frozen, publicly pretrained HuBERT encoder. Its features were shown to help multi-speaker inversion in the speaker-independent inversion paper, and freezing it keeps raw audio from being needed anywhere but on site. The shared model is a bidirectional GRU in the style of that same paper, so architectural gains are not confused with federation gains.

Fixing the canonical space. The canonical space has a scale-and-offset ambiguity, because any change in $f_\theta$ could be absorbed by every $g_k$. We remove it in two ways. Each $g_k$ is regularized toward the identity. The per-channel mean and variance of the canonical trajectories are pinned to zero mean and unit variance, using statistics aggregated by the server.

Training schedule. Training alternates within each round. Clients first update $g_k$ with $f_\theta$ fixed, then update $f_\theta$ with $g_k$ fixed, and send only the change to $f_\theta$. This mirrors CAFT's alternation and its pre-training warm start.

Enrolling a new site. A new site with a few minutes of data fits only $g_k$, around a hundred parameters, against the frozen shared model. We test this on held-out speakers. We also test it on FROST-EMA (FROST-EMA: Finnish and Russian Oral Speech Dataset of Electromagnetic Articulography Measurements with L1, L2 and Imitated L2 Accents), which adds unseen languages (Finnish, Russian) and second-language speech.

Second modality, as a secondary test. We repeat the split on ultrasound-to-speech conversion with UltraSuite-TaL80 (Conformer-based Ultrasound-to-Speech Conversion). There the private static map sits on the input side, because probe placement and tongue anatomy distort the image. The authors of that paper attribute large per-speaker variance to exactly these factors and call for speaker adaptation. With only four speakers this is a feasibility check, not a headline result.

Baselines

Local-only training. The do-nothing reference trains one model per client on its own data with no collaboration. The single-corpus results in the multi-corpus inversion paper bound what this achieves. Native correlation is 0.826 on EMA-IEEE. Transfer is poor: the XRMB-trained model drops to 0.453 correlation when tested on EMA-IEEE. Our method must beat this clearly on cross-site and held-out speakers. Otherwise federation buys nothing.

Centralized multi-corpus training. This is the strongest existing method and the accuracy ceiling. It is the shared-body, per-dataset-head model of Multi-Corpus Acoustic-to-Articulatory Speech Inversion, which reports validation correlations of 0.768 (XRMB), 0.819 (EMA-IEEE) and 0.780 (MOCHA-TIMIT). A second centralized reference is the speaker-independent model of Speaker-Independent Acoustic-to-Articulatory Speech Inversion, which reports 0.784 correlation on its combined EMA data. We retrain both with our front-end, split and tuning budget, so the comparison does not rest on their reported numbers alone. "Beating" this ceiling means coming within 0.02 mean correlation while lowering the shared model's identifiability. We do not expect to exceed it. No reported result was found on Lacuna for federated acoustic-to-articulatory inversion, so there is no prior federated number to beat.

Plain FedAvg with per-corpus output heads. This is the obvious port of the centralized model to federation, and the most dangerous competitor. If it matches our method on accuracy and identifiability, the static-adapter idea adds nothing. In acoustic recognition, plain FedAvg trailed centralized training (16.33% vs 15.83% word error rate) in Federated Acoustic Modeling For Automatic Speech Recognition. We do not assume the same gap appears here.

CAFT-style input adaptation. This baseline applies a per-client affine transform to the HuBERT features, with the paper's pre-training warm start. In that paper the warm-started variant reached 15.13% word error rate, below the centralized baseline. That result makes it a serious contender for the claim that a private per-client transform is enough.

Ablations that isolate the key ingredient. The first gives $g_k$ temporal context, either as a small convolution or a GRU. If static versus temporal makes no difference to identifiability, our anatomical argument is wrong. The second moves $g_k$ from the output to the input. The third uses a FedSpeech-style private speaker embedding fed to the shared model. This tests unconstrained private capacity against structurally constrained private capacity.

Fairness controls. All federated variants use the same rounds, local epochs, client sampling and a matched hyperparameter search budget. Each method also gets the same number of on-site adaptation steps for enrollment.

Why not just…?

Why not just centralize the data? For XRMB, MOCHA-TIMIT and HPRC we can centralize, and we do, as the ceiling. The case for federation rests on data that cannot be pooled: paediatric therapy recordings like UltraSuite (UltraSuite: A Repository of Ultrasound and Acoustic Data from Child Speech Therapy Sessions) and clinical cohorts like the Moebius-syndrome speakers in Acoustic similarities, articulatory uniqueness. Even for such data, centralizing the raw recordings is the bigger exposure, but releasing a centrally trained model is also an exposure. Our identifiability probes measure the second risk directly. If the centrally trained model turns out no more identifying than ours, the privacy argument reduces to the raw-data argument alone. We would report that.

Why not just fine-tune a public inversion model locally at each clinic? This is a real threat. A clinic could take the speaker-independent model and adapt it with no federation at all. Our learning curves compare that option against federated training at matched on-site data, from minutes to hours. Federation wins only if pooling motor dynamics across sites beats public data plus local tuning. The strongest evidence that it might is the cross-site collapse to 0.453 correlation in the multi-corpus paper. Our enrollment test on FROST-EMA settles the question for unseen languages.

Why not just add differential privacy to FedAvg? Differential privacy clips each update and adds calibrated noise, which gives a formal guarantee. With fewer than a hundred tiny clients, that guarantee usually costs a lot of accuracy. We run DP-FedAvg at several privacy budgets and plot accuracy against identifiability for every method on the same axes. If DP-FedAvg dominates our method on that plot, the static-adapter design is unnecessary. The two can also be combined, since a lower-identity shared model may need less noise.

Why not just use CAFT? CAFT's transform acts on acoustics, and acoustics hide articulatory differences: Moebius speakers match controls' formants while moving their tongues quite differently (Acoustic similarities, articulatory uniqueness). We expect an input-side transform to leave anatomical variation in the articulatory targets. The input-versus-output ablation tests this directly.

Proposed Experiments

Accuracy. We report Pearson correlation and RMSE per tract variable on held-out utterances of training speakers, on held-out speakers, and across corpora. The cross-corpus protocol reuses the one from the multi-corpus inversion paper, so its 6.9% and 29.5% cross-dataset gains are directly comparable.

Identifiability. Two attacks probe the shared model. First, a speaker-identification classifier is trained on the shared model's canonical trajectories, and on its hidden states, for held-out utterances of training speakers. We report its equal error rate and log-likelihood-ratio cost. The size-and-shape tongue features of the vowel-production biometrics study (5.6% error) give a reference for how identifying raw articulation is. Second, a membership-inference attack asks whether a speaker was in training given the released shared model. As a sanity check, the same speaker probe applied to the private adapters $g_k$ should be highly identifying. That would confirm that identity has moved into the private part rather than vanished.

Decisive check. The proposal succeeds if three things hold together. Our method stays within about 0.02 correlation of centralized training. Its shared model is clearly less identifying than plain FedAvg and the centralized model. The temporal-adapter ablation loses that identifiability advantage. If the third fails, anatomy is not what the static adapter captures. If the first fails, the constraint costs too much. Either outcome is informative for anyone planning multi-clinic articulatory studies.

Secondary check. On TaL80 we report per-speaker mel-cepstral distortion with an input-side static adapter, plus a small MUSHRA listening test. The conformer paper found objective and perceptual scores disagreed, so both are needed.