Diffusing the Vocal Tract, Not the Video

Generating a real-time MRI video of someone's vocal tract from their voice is a useful tool for phonetics, clinical speech assessment, and articulatory speech synthesis. Such a video is a moving side view of the throat, tongue, jaw and lips. The current approach uses a diffusion model, a generator that learns to turn random noise into samples by removing noise step by step. These models denoise a full stack of image frames conditioned on speech features, and they are slow because every one of dozens of denoising steps runs a large 3D network over every pixel of every frame.

Most of those pixels encode nothing that changes. The skull, the palate, the spine and the scanner background stay fixed. Speech physiology says the moving part is small. The vocal tract is well summarized by an aperture function, which gives the air gap at roughly 54 to 62 sections from larynx to lips (Lee et al.), plus a few articulator points. We propose to run diffusion only on this compact physiological state, conditioned on a per-speaker rest frame, and to render pixels with a cheap deterministic network only when pixels are needed. The central question is how much quality per unit of compute the physiological representation buys.

Core Claim

A diffusion model over per-frame aperture functions and articulator contours will match or beat pixel-space speech-to-MRI diffusion on video realism, and will be clearly better on tongue-to-palate contact events. It will do this at a small fraction of the sampling FLOPs. The contact advantage has a concrete source. When aperture is represented as a non-negative quantity that can hit exactly zero, a closure for /t/, /d/ or /n/ becomes an explicit, sharp event. It no longer has to emerge as a pixel boundary from noise.

The claim that is new, and that can be falsified, is about compute. The same parameter count may achieve two different quality-versus-compute curves depending on the representation. A representation with physiological structure should dominate a generic learned latent of the same dimension. If it does not, the physiology is decoration and the gain is only compression.

Where This Sits

Two papers motivate the proposal by exposing both the cost and the failure mode. Speech2rtMRI generates midsagittal rtMRI video with a 3D diffusion U-Net conditioned on WavLM features and a single rest frame of the speaker. It reports FVD 1420 on 15 unseen USC speakers, and phoneticians noted jitter at tongue–palate contacts. Speech2-MRI runs a 3D-UNet in a VAE latent of MRI frames with a PNDM sampler. It reports FVD 538.72 against 984.46 for a VQGAN baseline, and it too struggles with alveolar closures.

Both papers diffuse image-like tensors. We diffuse physiological coordinates and reuse their conditioning, meaning WavLM features and a rest frame. Speech2-MRI's VAE latent is the closest thing to our idea, and it becomes our key ablation. It compresses the pixels but carries no physiological meaning, so it cannot express contact as a hard zero.

A third paper, Optical Flow Guided Tongue Trajectory Generation, adds a Lucas-Kanade trajectory loss to ultrasound tongue diffusion. Cost limited it to isolated words. It shows that motion-aware objectives help, but it still pays pixel-space cost and sees only the tongue surface.

The efficiency literature supplies the method. Würstchen cut training from about 200,000 to 24,602 A100-GPU hours by diffusing a compact semantic latent. We apply the same logic, with physiology choosing the latent instead of a learned encoder. DiffProsody shows that a 4-step adversarial diffusion suffices for low-dimensional speech prosody vectors. Conditional flow matching (Pan et al.) and consistency models (Zhang et al.) give us few-step samplers.

The physiology literature constrains the design. Aperture functions from rt-MRI carry enough affect information to separate emotions (Lee et al.), and contour curves classify affect about as well as acoustic features (Narayanan et al.). This tells us the compact state is not discarding the expressive signal.

VocalTrax inverts audio into physical tract parameters by per-utterance gradient descent through a differentiable tube model. It is physically grounded but not amortized. We amortize the inversion and do not simulate acoustics.

Research Program

Building the targets. First we build the physiological targets for the USC 75-speaker rtMRI corpus. We segment air–tissue boundaries with a segmentation network trained on a hand-annotated subset. From the boundaries we compute the aperture along a fixed grid of lines perpendicular to the tract midline, following the gridline method of Lee et al.. We also keep a small set of boundary landmarks: tongue tip, tongue dorsum, velum, lips, and larynx height. Larynx height captures tract-length changes. Each frame becomes a vector of about 70 to 90 numbers. The static anatomy is the palate and pharyngeal wall shape at rest, and it is encoded once per speaker from the rest frame.

The generator. The generator is a small 1D temporal transformer that denoises the whole utterance's state sequence, conditioned on WavLM features and the speaker's rest-frame embedding. Aperture is produced through a non-negative output with a separate closure logit per section. This makes full contact a representable value rather than a limit the model must approach.

We train three sampler families on the same network: a standard multi-step diffusion sampler, a conditional flow matching model sampled with 1 to 8 steps, and a consistency-distilled one-step model. Diffusion is justified here because acoustic-to-articulatory inversion is one-to-many. Several tract shapes produce the same sound, for example lip rounding trading against larynx lowering. Whether that stochasticity actually matters is tested directly against a deterministic regressor, described below.

Pixel evaluation. For evaluation in pixel space, a per-speaker renderer turns the rest frame plus the state sequence into MRI frames. The renderer is a deterministic warping network trained with a reconstruction loss. Its cost is always counted in our compute totals.

Compute accounting. We report sampling FLOPs per second of generated articulation, wall-clock time on a single GPU, and training GPU-hours. This gives a full quality-versus-compute curve for each method rather than a single operating point.

Baselines

Naive reference points. The weakest reference is the speaker's rest frame, or the speaker's mean aperture, repeated for the whole utterance. Any method must clearly beat this on contact and trajectory metrics. Otherwise those metrics are only measuring static anatomy.

Strongest existing method. The strongest competitor is Speech2rtMRI. It targets exactly this task on exactly this corpus and split, uses the same WavLM and rest-frame conditioning, and reports FVD 1420 on the 15 unseen speakers. We reimplement it on the 60/15 split and train it with the same GPU budget as our model. We evaluate it at its native step count and at reduced step counts, so it has its own quality-versus-compute curve. The file gives no reported sampling latency or FLOPs for Speech2rtMRI or Speech2-MRI; no reported result found on Lacuna for either, so we measure them ourselves.

For reference, Speech2-MRI reports FVD 538.72. Its evaluation included different data, including a clinical cohort, so its number is not directly comparable. We will not claim to beat it unless we rerun it on our split. Its VAE-latent design is instead re-run as the latent ablation below.

What counts as winning. Beating Speech2rtMRI means three things together. First, FVD on unseen speakers equal to or better than our reimplementation. Second, a higher closure F1 on coronal stops. Third, at least an order of magnitude fewer sampling FLOPs at the matched quality point.

Ablations isolating the key ingredient. The ablation that isolates physiology is a learned-latent diffusion model. We train an autoencoder on MRI frames with a latent of the same per-frame dimension as our state vector, then diffuse that latent with the same network and sampler. This holds dimensionality and compute fixed while removing the physiological structure. A second ablation removes the closure logit and non-negativity. This tests whether the contact advantage comes from representing contact explicitly or merely from having few dimensions.

Why not just…?

Why not just distill the pixel-space model to a few steps? Few-step samplers do reduce cost, and DiffProsody shows 4 steps can work for speech representations. But distillation shrinks the number of steps, not the cost of each step. Each step still runs a 3D network over every pixel of every frame. Our experiment compares the pixel model at 1 to 8 steps against our model at the same step counts. The pixel model could close the gap on FVD. If it does, our contribution narrows to the contact results and to physiologically readable outputs, and we will report it that way.

Why not just use a compact learned latent, as Speech2-MRI and Würstchen do? This is the real threat. Much of our FLOP saving could come from dimensionality alone. The matched-dimension latent ablation settles it. Physiology earns its place only if our model beats that latent on contact F1 and trajectory error at equal compute. An FVD-only win would not be enough. If the learned latent ties, we will conclude that compression, not physiology, is doing the work.

Why not just regress the aperture sequence from audio with a deterministic network? For mean trajectory error, a regressor may do as well as diffusion, and at the lowest cost. The one-to-many argument predicts that a regressor will average compensatory configurations. That averaging should show up in two ways. Closures would come out shallow, since averaging a zero with small non-zero gaps is non-zero. The spread of generated shapes across repeated samples would also be too narrow compared with real repetitions of the same sentence. We therefore include a regressor in the same representation. If it matches diffusion on closure F1 and distributional metrics, we will recommend it over diffusion. Showing when diffusion's cost is not needed is itself part of the compute question.

Proposed Experiments

All training and testing uses the USC 75-speaker rtMRI corpus with the 60 training and 15 unseen-speaker split from Speech2rtMRI.

Scoring. One boundary extractor scores every method: a single segmentation network plus the gridline aperture computation. It is run on both real and generated frames, so pixel-space baselines are judged in physiological coordinates too, and extraction noise affects all methods equally. The metrics are FVD on rendered or generated video, aperture RMSE per tract region, and closure detection F1 with onset-timing error for /t/, /d/ and /n/ against the real video. We also measure how varied the generated shapes are across repeated samples. Compute is reported as sampling FLOPs, wall-clock time, and training GPU-hours.

Decisive check 1: compute. Plot FVD and closure F1 against sampling FLOPs for five systems: the pixel diffusion model, the learned-latent ablation, our physiological model, the regressor, and the static reference. The proposal succeeds if the physiological curve lies above both the pixel and learned-latent curves across the low-compute range.

Decisive check 2: contact. Closure F1 on coronal stops should improve over the pixel model and over the variant without the closure logit. This directly addresses the jitter reported by the Speech2rtMRI authors.

Decisive check 3: physiological plausibility (secondary and weaker). We condition the unchanged model on emotional read speech from ESD, the Emotional Speech Database (Emotional Voice Conversion: Theory, Databases and ESD). We then ask whether the generated aperture functions reproduce two findings from Lee et al.. The first is the dynamism ordering: anger 844.5, sadness 691.5, happiness 557.7, neutral 516.3. The second is the shorter active tract in happy speech, 54 to 58 sections against 60 to 62. No emotional rtMRI data are available to us for training, so this is a test of plausibility, not accuracy. A failure would still be informative: it would mean that a model trained on neutral speech does not transfer expressive tract behavior, which bounds what compact physiological diffusion can offer for affective synthesis.