Abstract
Real-time speech-to-avatar synthesis often neglects the complex dynamics of the inner mouth due to the high latency associated with deep articulatory models. This paper presents a low-latency pipeline that animates the tongue and teeth in real-time by leveraging acoustic-to-articulatory inversion (AAI) and a rolling context window. The system achieves an average latency of approximately 130ms while maintaining high anatomical accuracy, making it suitable for applications like pronunciation training and virtual embodiment.
Introduction
While speech-driven avatars are becoming common, most high-quality systems operate offline, processing pre-recorded audio. Real-time "streaming" versions often struggle with two primary factors: latency and anatomical detail. Specifically, most avatars focus on external facial expressions and lip-syncing, ignoring the complex movements of the tongue and inner mouth.
Accurate inner-mouth animation is crucial for applications like second-language (L2) pronunciation training—where students need to see where to place their tongue—and for providing a sense of "virtual embodiment" for paralyzed patients. The challenge lies in the fact that deep learning models usually require a few seconds of audio context to accurately predict speech movements, which is inherently at odds with the "instant" requirements of a live stream.
Methodology
The researchers proposed a multi-stage pipeline to bridge the gap between raw audio and a 3D avatar.
Acoustic-to-Articulatory Inversion (AAI)
The system takes 0.1-second batches of audio and predicts Electromagnetic Articulography (EMA) features. These are 12-dimensional coordinates representing the midsagittal positions of the tongue tip (tt), body (tb), dorsum (td), lips (ul, ll), and incisors (li).
Model Architectures and Context Management
Two architectures were tested: a BiGRU-based model using HuBERT features and a Transformer-based model using WavLM features. The Transformer model proved more accurate. To solve the "context vs. latency" problem, the system prepends the current 0.1s audio batch with the previous 1–2 seconds of audio. This rolling context window provides the history needed for accurate predictions without waiting for future audio. A 1-second window emerged as the "sweet spot," providing enough linguistic context for co-articulation while keeping total lag around 133ms.
3D Rigging and Optimization
The predicted EMA coordinates are mapped to a joint-based 3D model in Autodesk Maya. To maintain smooth animation, Cubic Bezier curves interpolate between batches, preventing jitter at boundaries. To minimize data transfer lag, the team implemented a shared memory buffer, allowing the AI model and the 3D renderer to communicate almost instantaneously.
Results
The models were trained and evaluated using the Haskins Production Rate Comparison (HPRC) dataset, focusing on the "M02" speaker.
Latency Profiling
The system achieves an average streaming latency of 130ms for every 0.1 seconds of audio. Profiling identified the AI model as the primary bottleneck (~77ms), followed by the 3D rendering engine (~56ms). Data transfer ("sending") was negligible (~4ms) due to the shared memory approach.
Accuracy and Qualitative Performance
The Transformer-based model achieved a 0.792 Pearson correlation with ground-truth articulatory movements. Qualitatively, the avatar accurately portrays complex tongue movements, such as the tongue tip nearing the lower incisor for "ah" sounds or receding for "ooh" sounds.
Discussion and Limitations
A primary limitation is the initialization phase. The first 0.5 to 1.0 seconds of a stream are often noisier because the rolling context window is not yet full. The researchers found that padding the start with a recording of a generic vowel sound significantly improved accuracy compared to using silence.
Future work aims to eliminate the need for manual initialization and to improve transduction techniques. Additionally, while the current model is trained on a single speaker, moving toward a speaker-independent model is the next logical step for a general-purpose avatar tool.
Summary updated