Tokenizing Speech in Vocal-Tract Coordinates

When speech is turned into discrete tokens so that a language model can read and write it, most of what makes a voice expressive is thrown away. Emotion, emphasis, whispering and smiling survive only weakly. The usual tokens come from clustering the internal features of a self-supervised speech model, a network pretrained to predict masked stretches of audio. Clustering groups frames by which speech sound they are. It does not group them by how the sound was said. The alternative, a neural audio codec (a compressor trained to reconstruct the waveform), keeps pitch but memorizes the speaker's voice and does poorly on emotion. We propose a third option. First, recover from audio how the tongue, lips and jaw are moving, together with pitch, loudness and voicing. This step is called acoustic-to-articulatory inversion. Then tokenize in that low-dimensional physical space, which is already normalized across speakers.

The bet is this: expressive speech is partly a motor act. People open the jaw wider when shouting, spread the lips when smiling, and exaggerate vowels when enunciating. A tokenizer that quantizes those motions directly should keep paralinguistic information that clustering discards, and it should stay free of speaker identity in a way codecs are not. If that holds, articulatory tokens would close more of the emotion-recognition gap left by current discrete units. They would also carry speaking style through voice conversion without the oracle style labels that current systems need.

Core Claim

Quantizing speech loses paralinguistic information for two reasons. Clustering in a high-dimensional space spends almost all its centroids on phonetic distinctions. The space also mixes speaker, content and style together, so they cannot be separated. We claim that a small, factored, physically defined coordinate system reduces both problems. That system has 12 articulator trajectories mapped onto a single reference speaker, plus separate pitch, loudness and periodicity channels. Each stream can be quantized at its own resolution, and the speaker has already been factored out. The claim can be falsified in a specific way. If an equally small but physically arbitrary projection of the same encoder (for example its top 12 principal components) does as well, then the gain comes from low-dimensional factored quantization and not from articulatory grounding. We would report it that way.

What Exists and What Is Missing

The method comes mainly from Coding Speech Through Vocal Tract Kinematics (SPARC). SPARC fits a linear probe on frozen WavLM Large to predict 12 electromagnetic articulography (EMA) traces from one speaker in the MNGU0 corpus. It adds pitch and loudness, plus a 64-dimensional speaker embedding, and drives a modified HiFi-GAN vocoder. It reaches 5.43% WER on LibriTTS-R and 3.73% on unseen VCTK. SPARC stays continuous and is evaluated only on intelligibility and timbre, never on emotion or style. We reuse its inversion and vocoder, discretize its streams, and measure exactly what it left out.

Self-Supervised Models of Speech Infer Universal Articulatory Kinematics motivates the approach. It shows that linear probes on self-supervised layers predict EMA with correlation above 0.8 across 62 speakers, against about 0.66 for spectral features. It also constrains us: the data is controlled read speech, and EMA misses the glottal source. That is why our token set must include its own source channels, and why we cannot assume inversion stays faithful on whispers or laughter. The entropy of the articulatory phonological code showed that tract variables can be discretized into 181 gesture classes, but only on synthetic trajectories. Ours would be the first articulatory discretization built from natural audio.

On the loss side, Recovering Performance in Speech Emotion Recognition from Discrete Tokens is the closest work and our main competitor. It measures a 6–14% Macro F1 drop on MSP-Podcast when WavLM features are discretized. It recovers part of the gap by fusing all 24 layers and by adding discretized OpenSMILE pitch, formant and intensity features, reaching Macro F1 0.3534. It also finds little gain beyond K=1,000. Its added features are surface acoustic statistics: they are not speaker-normalized and cannot drive resynthesis. Expresso defines the generative side of the problem. HuBERT units allow voice conversion (about 7.6% WER when swapping speakers) but keep style only 28.5% of the time unless an explicit style ID is given (81.4%). EnCodec preserves pitch (frame error 0.04) but memorizes the speaker.

The audio-LLM diagnoses in Heard but Not Heeded and Do Audio LLMs Listen or Read? support the broader motivation. Encoders hold style information that later layers discard. In particular, the first paper reports that adversarial disentanglement of style from content fails. This is evidence that the separation must be built into the representation's coordinates rather than trained in afterward. Their fixes patch continuous models, while we change the tokens themselves.

Research Program

We build three token streams at 50 Hz from frozen WavLM Large audio. The articulatory stream is the SPARC 12-channel kinematic output, quantized with a learned vector-quantization codebook. We also test a product quantizer over articulator groups (lips, tongue tip, tongue body, jaw) so that, for instance, a lip-spreading change does not need its own joint centroid. The source stream quantizes three channels:

  • log pitch, normalized per speaker so the token encodes contour rather than register;
  • loudness;
  • a periodicity channel taken from the pitch tracker's voicing confidence. SPARC lacks this channel, and without it whispered speech is unrepresentable.

The speaker embedding stays continuous and global, as in SPARC, and is not part of the token sequence. The tokens are decoded by retraining SPARC's FiLM-conditioned HiFi-GAN on the quantized inputs.

All comparisons are made at matched bitrate in bits per second. A WavLM k-means stream at K=1,000 and 50 Hz is roughly 500 bps. We therefore sweep a small set of operating points, and compare every system on a curve of paralinguistic score against bitrate rather than at a single point.

Before the main experiments, we address whether inversion survives expressive speech (Gap 4 in the file). Expresso has no articulatory ground truth, so we use two indirect checks. The first is resynthesis WER per style on continuous SPARC. The second is coding-recoding consistency per style: encode, resynthesize, re-encode and correlate, against SPARC's reported overall similarity of 0.944. Styles where consistency collapses, plausibly whisper and laughter, mark where the articulatory stream cannot be trusted. We report them separately rather than averaging them away.

Baselines

The naive reference points come first. One is continuous WavLM features with the same downstream classifier, which is the ceiling discretization is measured against. The other is single-layer WavLM k-means at K=1,000, the floor that Chung et al. start from. We rerun both ourselves on MSP-Podcast v1.12 with the same splits, so that "fraction of gap recovered" is computed on our own numbers.

The strongest existing method on emotion recognition is the augmented discrete system of Recovering Performance in Speech Emotion Recognition from Discrete Tokens. It fuses all 24 WavLM layers with discretized formant features and reports Macro F1 0.3534 on MSP-Podcast, about 75% of the discretization gap. We reproduce it using the same downstream classifier, epochs and hyperparameter search budget as our tokens. Beating it means one of two things. Either articulatory-plus-source tokens alone exceed its Macro F1 at equal or lower bitrate, or adding our streams to its fused tokens exceeds 0.3534 by more than the seed-to-seed spread across five runs. A gain within that spread counts as a tie.

On style-preserving resynthesis, the reference is Expresso's HuBERT-unit voice conversion: style accuracy around 28.5% without a style ID and 81.4% with an oracle ID. We rerun it with the same style classifier we apply to our outputs. We also run EnCodec at matched bitrate. Its reported pitch error of 0.04 frames is the target for our source stream, and its speaker leakage is the target we must beat on speaker-verification similarity to the source speaker. Continuous, unquantized SPARC serves as the ceiling for our own representation.

The ablations isolate the key ingredient:

  • source tokens only, which is essentially discretized prosody and close in spirit to OpenSMILE augmentation;
  • articulatory tokens only;
  • WavLM k-means plus our source tokens, which tests whether the articulatory stream adds anything beyond better prosody;
  • a 12-dimensional principal-component projection of the same WavLM layer, quantized identically but with no articulatory supervision;
  • the same projection without speaker normalization.

Why not just…?

Why not just use a neural audio codec, which already keeps pitch and detail? The file reports that SpeechTokenizer, EnCodec and DAC score well below discrete WavLM tokens on MSP-Podcast emotion recognition (Chung et al.). Expresso also shows EnCodec memorizes the speaker and cannot support voice conversion. Codecs fail at the separation we want, so they serve as a matched-bitrate baseline rather than a substitute.

Why not just raise the k-means codebook size? Chung et al. already find marginal gains past K=1,000 on dense WavLM configurations, with a residual gap remaining. Our bitrate curves include large-K k-means, so the result is shown directly and does not rest on argument.

Why not just add OpenSMILE features or fuse layers? This is the real threat. Pitch and intensity carry much of emotion, and the source-only ablation may capture most of our gain. If so, the articulatory stream would be shown redundant for classification. Its remaining value would then lie in resynthesis: it is speaker-normalized and decodable, which OpenSMILE statistics are not. The Expresso voice-conversion test settles that. The decisive comparison is WavLM k-means plus our source tokens against the full articulatory system.

Why not just quantize any small projection of WavLM? Physical grounding may not matter; a low-dimensional factored code might be enough. The 12-dimensional principal-component control answers this. If it matches the articulatory stream, we drop the grounding claim and report the factorization result.

Proposed Experiments and Decisive Checks

The first decisive check is on MSP-Podcast: at roughly matched bitrate, does the articulatory-plus-source tokenizer recover more than the 75% of the discretization gap reported by the formant-augmented system? Does the articulatory stream add accuracy over WavLM units plus source tokens? A positive answer to both supports the motor-act hypothesis. A positive answer to only the first says the win is prosody quantization.

The second check is on Expresso voice conversion without style IDs. Does style accuracy rise well above HuBERT units' 28.5% toward the oracle 81.4%, while speaker similarity to the source stays near that of HuBERT units rather than EnCodec's? We expect per-style gains on enunciated, happy and smiling-type styles, where articulation should matter. We expect smaller gains on whisper unless the periodicity channel does its job. We will check this split explicitly, because it is the most specific prediction the articulatory account makes.

Secondary metrics are resynthesis WER and pitch frame error against EnCodec's 0.04, and the per-style inversion consistency described above. Together these show where the physical coordinate system holds and where it breaks.