Real-time MRI of the vocal tract shows how the tongue, lips, velum and larynx move during speech. Much of the existing data was recorded too slowly. Large legacy corpora run at about 23 frames per second, yet Töger et al. reconstructed the same speech at several frame rates and found that roughly 42 fps is the minimum for measuring fast tongue-tip motion without blur. At 23 fps, taps, stop closures and releases fall between frames. No one can go back and rescan those speakers at a higher rate. Separately, a few newer corpora recorded other people at about 83 fps. This proposal asks whether we can learn "what fast articulation looks like" from those fast recordings of other speakers and use it to fill in the missing motion in slow recordings. The key constraint is that the filled-in motion must be exactly consistent with what the slow scanner actually measured.
Concretely, a generator network takes a slow sequence and outputs a fast one. Two losses shape it. A physics loss requires that the output, pushed back through the known slow-acquisition process, reproduce the recorded measurements; in MRI, a slow frame is literally a sum of the raw frequency-space samples collected over a time window. An adversarial loss, from a discriminator network trained to tell real from generated clips, requires that short clips of the output's articulator motion be indistinguishable from real 83 fps clips of other speakers. The prior's only job is to decide what happens inside each time window, the part the measurements cannot see. It is never asked to decide what the measurements already determine.
Core Claim
The missing within-window motion in slow speech MRI should come from an unpaired, learned prior over fast articulator motion, tied to the exact temporal-integration forward model. Two established alternatives supply it differently, and both make known errors on speech:
- Temporal smoothness penalties. These are what compressed sensing uses. Töger et al. show that they act as low-pass filters and systematically blunt peak velocities and displacement extremes.
- Symmetry assumptions. These are what fully unsupervised dynamic MRI uses. DDEI enforces equivariance to temporal shifts, time reversal and smooth warps. That design was validated on near-periodic cardiac cine. Speech is aperiodic, and closures and releases are not mirror images, as the directional asymmetry reported by Constrained LDDMM indicates.
The testable prediction follows directly. Relative to both alternatives, the unpaired prior should recover peak tongue-tip velocities and brief contact events that the others either blunt or miss. It should do this without inventing contacts that the measurements rule out.
Relation to Prior Work
The method comes from OT-cycleGAN. That work removed CycleGAN's backward generator (the second network that maps outputs back to inputs) and replaced it with the known MRI undersampling operator. With 4x acceleration on HCP brain data, the unpaired model reached 37.73 dB PSNR, against 37.32 dB for a supervised baseline. That result was obtained on static 2D slices with a fixed Cartesian sampling mask. We extend the same move to a time-varying spiral operator that integrates many fast frames into one slow measurement. We also draw the adversarial prior from a different corpus and a different set of speakers.
DUNCAN motivates guarding against structural hallucination in unpaired MRI translation. It works purely in the image domain and on static anatomy, whereas our physics loss pins every output to the measurements.
The closest speech-side work is Generating high-resolution 3D real-time MRI of the vocal tract. It fuses slow 2D video with static 3D volumes through Hungarian matching and non-negative linear combinations of basis shapes. It improves spatial detail, not temporal resolution, and cannot represent postures missing from its basis. Constrained LDDMM morphs smoothly between two static shapes under 72 fps contour constraints. It needs fast data at inference time, which our target corpora do not have.
Two self-supervised lines constrain the design:
- Measurement splitting. SSDU and ZS-SSL hold out part of the acquired frequency-space data to train or validate a reconstruction without ground truth. We reuse ZS-SSL's held-out validation subset to tune the loss weight and stop training.
- The "complexity paradox." Acar et al. found that explicit optical-flow modules collapse under self-supervision because aliasing corrupts the flow estimates. We therefore use a compact spatiotemporal generator with no explicit flow estimation.
Research Program
Write $x = (x_1,\dots,x_T)$ for the unknown fast image sequence at about 83 fps, and $y_j$ for the raw frequency-space data behind slow frame $j$. With $\mathcal{F}$ the Fourier transform, $C$ the coil sensitivities and $P_t$ the spiral interleaf sampled at fast time $t$ within window $W_j$, the acquisition is
$$y_j = \sum_{t \in W_j} P_t \mathcal{F} C x_t + \epsilon_j ,$$
where $\epsilon_j$ is measurement noise. The generator $G_\theta$ maps the slow data to a fast sequence and is trained with
$$\min_\theta \; \sum_j \Big\| \sum_{t\in W_j} P_t \mathcal{F} C\, [G_\theta(y)]_t - y_j \Big\|^2 \;+\; \lambda\, \mathcal{L}_{\text{adv}}\big(\phi(G_\theta(y)),\, \phi(x^{\text{fast}})\big).$$
Here $x^{\text{fast}}$ are real clips from other speakers' 83 fps corpora. $\phi$ is a fixed feature map, and $\lambda$ trades measurement fidelity against realism.
The main design choice is $\phi$. A discriminator that looks at raw pixels would pull the output toward the scanner, coil and contrast of the fast corpora. We instead let the discriminator see articulator contour trajectories, extracted by a frozen segmenter trained on the expert masks of USC-75 (VocSegMRI). The prior then constrains motion, not appearance. The pixel-level discriminator is kept as an ablation.
$\lambda$ is chosen without ground truth, using a ZS-SSL-style held-out set of acquired interleaves (ZS-SSL).
When raw spiral data are unavailable and only sliding-window video exists, the forward model reduces to image-domain temporal integration over each window. This is an approximation, and its cost is measured below.
Fast-motion clips come from USC-TIMIT at about 83 fps (Toutios & Narayanan), the IPA resource at 83.33 fps (Toutios et al.) and USC-75 at 83.28 fps. The slow targets are MRI-TIMIT, acquired at 12.5 fps and reconstructed to 23.18 fps (Narayanan et al.), and the 17-speaker morphology database at 23.18 fps (Database of Volumetric and Real-Time Vocal Tract MRI).
Baselines
Naive reference. The slow sliding-window reconstruction, with frames linearly interpolated in time up to 83 fps. It shows how much kinematic error comes purely from slow sampling.
Classical compressed sensing. The same raw data are binned at 83 fps and reconstructed with temporal total variation in BART, following Töger et al.. $\lambda$ is swept over their published range [0.00025, 0.012]. We report the best setting chosen by the same held-out-measurement rule our method uses, and also the oracle-best setting. This is the strongest training-free competitor, and it isolates the blunting effect of smoothness penalties.
Unsupervised dynamic reconstruction (DDEI). DDEI is run at 83 fps with our generator architecture, our forward operator and the same training budget. The comparison therefore isolates symmetry priors against a learned motion prior. Its reported result is 33.3 dB PSNR on CMRxRecon cardiac cine at 8x acceleration, against 36.4 dB for supervised training and 17.1 dB for temporal splitting (t-SSDU). There is no reported result on Lacuna for DDEI, or any other method, on speech rtMRI temporal upsampling.
Supervised training on synthetic pairs. The 83 fps corpora are degraded through the same forward operator, and a network with the same architecture is trained supervised on those pairs. This is the most dangerous baseline and is discussed below.
Standard CycleGAN. Two generators with a pixel discriminator and cycle loss, as the canonical unpaired approach.
Ablations. (1) No adversarial term, which reduces to data consistency plus architecture prior. (2) Pixel discriminator instead of contour-trajectory discriminator.
What counts as beating a baseline. On speaker-held-out retrospective tests, our method must significantly reduce error in peak tongue-tip velocity and constriction duration, paired by utterance. It must do so with a false-contact rate no higher than the compressed-sensing baseline, and it must beat synthetic-pair supervision on the cross-corpus test.
Why not just…?
Why not just train a supervised model on synthetic pairs made by degrading 83 fps data? This is a real threat. If the degradation model and the scanners matched perfectly, it could equal our method. It fails when the slow target corpora differ in field strength, coil array, speaker population or reconstruction pipeline, which is the situation for legacy data. Our method trains on the target corpus's own measurements and borrows only motion statistics from elsewhere. Experiment 2 settles the question by training both methods on one set of corpora and testing on another. If synthetic-pair supervision matches us across corpora, the unpaired component is unnecessary, and we would report that.
Why not just re-reconstruct the raw data at a higher frame rate with compressed sensing? At 12.5 fps acquisition, an 83 fps bin holds only a few spiral interleaves. Recovery then depends heavily on the regularizer, and Töger et al. show that temporal regularizers trade blunted peaks against aliasing that breaks segmentation. The BART sweep tests whether any setting of $\lambda$ escapes this trade-off.
Why not just use DDEI's equivariance with no external corpus? DDEI needs no second dataset, which is attractive. Its time-reversal symmetry, however, assumes that closures and releases look alike, and the LDDMM asymmetry result suggests they do not. The head-to-head comparison, broken down by gesture direction (closure versus release), tests this directly.
Why not just apply video frame interpolation? Generic interpolators synthesize motion between two frames. Each slow MRI frame is not a snapshot but a blurred integral over its window, so interpolating between frames discards the within-window evidence. The naive-interpolation reference bounds this error.
Proposed Experiments
Experiment 1: retrospective recovery. From 83 fps raw data of held-out USC-TIMIT or IPA speakers, we rebin interleaves to emulate a 12.5 fps acquisition. The prior is trained only on other speakers. The 83 fps reconstruction of the same raw data serves as the reference. Because that reference is itself regularized, we compare kinematics rather than pixels:
- error in peak tongue-tip velocity and displacement extremes;
- error in closure onset and duration;
- detection rate of taps and flaps in the IPA material;
- Dice and HD95 against expert USC-75 masks on frames that fall between slow samples.
For context, VocSegMRI reports Dice 0.95 and HD95 4.26 mm on full-rate frames, an upper reference for segmentation quality. The false-contact rate counts closures that appear in the output but not in the reference.
Experiment 2: cross-corpus transfer. Both our method and the synthetic-pair baseline are trained with prior data from USC-TIMIT alone. They are tested on degraded IPA and USC-75 data, which come from different speakers, sessions and reconstruction settings. This is the decisive test between the two methods.
Experiment 3: legacy validation with an independent signal. MRI-TIMIT has no fast reference, so we use the synchronized audio, which is never seen in training. Stop-closure and burst times estimated from the upgraded video are compared with acoustic landmarks from SailAlign alignments. The claim is supported only if image-derived timing agrees with acoustic timing better than timing from the slow video does, and better than timing from the compressed-sensing and DDEI outputs.
Experiment 4: hallucination stress test. Slow data of utterances with no alveolar contact are passed through the model with a prior trained heavily on tap-rich IPA clips. Any inserted contact is a failure. We also report how the fraction of variance in each output lying outside the span of the measurements changes as $\lambda$ grows.
Experiment 5: forward-model approximation. The image-domain temporal-integration approximation is run on data where raw frequency-space data exist, and compared with the exact spiral operator. This measures the accuracy cost of processing corpora that lack raw data.
References
- Sensitivity of Quantitative RT-MRI Metrics of Vocal Tract Dynamics to Image Reconstruction Settings
- Fully Unsupervised Dynamic MRI Reconstruction via Diffeo-Temporal Equivariance
- Constrained LDDMM for Dynamic Vocal Tract Morphing: Integrating Volumetric and Real-Time MRI
- Unpaired Deep Learning for Accelerated MRI Using Optimal Transport Driven CycleGAN
- Learning MRI artefact removal with unpaired data
- Generating high-resolution 3D real-time MRI of the vocal tract
- Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data
- Zero-Shot Self-Supervised Learning for MRI Reconstruction
- Self-supervised Dynamic MRI Reconstruction
- VocSegMRI: Multimodal Learning for Precise Vocal Tract Segmentation in Real-time MRI
- Advances in real-time magnetic resonance imaging of the vocal tract for speech science and technology research
- Illustrating the Production of the International Phonetic Alphabet Sounds Using Fast Real-Time Magnetic Resonance Imaging
- A Multimodal Real-Time MRI Articulatory Corpus for Speech Research
- Database of Volumetric and Real-Time Vocal Tract MRI for Speech Science