Abstract
Articulatory trajectories like electromagnetic articulography (EMA) provide a low-dimensional representation of the vocal tract filter and have been used as natural, grounded features for speech synthesis. Differentiable digital signal processing (DDSP) is a parameter-efficient framework for audio synthesis. Therefore, integrating low-dimensional EMA features with DDSP can significantly enhance the computational efficiency of speech synthesis. In this paper, we propose a fast, high-quality, and parameter-efficient DDSP articulatory vocoder that can synthesize speech from EMA, F0, and loudness. We incorporate several techniques to solve the harmonics / noise imbalance problem, and add a multi-resolution adversarial loss for better synthesis quality. Our model achieves a transcription word error rate (WER) of 6.67% and a mean opinion score (MOS) of 3.74, with an improvement of 1.63% and 0.16 compared to the state-of-the-art (SOTA) baseline. Our DDSP vocoder is 4.9x faster than the baseline on CPU during inference, and can generate speech of comparable quality with only 0.4M parameters, in contrast to the 9M parameters required by the SOTA.
Introduction
Articulatory synthesis involves generating speech from the physical movements of human articulators, such as the tongue, lips, and jaw, typically measured via electromagnetic articulography (EMA). This technology holds significant promise for medical applications, including assistive devices for patients with vocal cord disorders or brain-to-speech decoding interfaces. By using physical features, these systems offer greater interpretability and control compared to traditional text-to-speech models.
However, current state-of-the-art (SOTA) neural vocoders, such as HiFi-CAR, are computationally intensive and possess large parameter counts. This high resource demand limits their deployment on edge devices like smartphones or medical implants. The objective of this research is to develop a vocoder that maintains high audio fidelity while being small and fast enough for real-time, low-footprint applications.
Methods
The proposed system utilizes a Differentiable Digital Signal Processing (DDSP) framework. Unlike standard neural vocoders that learn to generate raw waveforms directly, DDSP integrates a neural network "encoder" with traditional, interpretable DSP modules.
Key technical innovations include:
- Harmonic-plus-Noise (H+N) Model: Speech is decomposed into periodic harmonics (voiced components) and filtered noise (unvoiced components).
- Enhanced Expressivity: The model employs both sine and cosine harmonics to better approximate a Fourier series, enabling richer tonal representation than standard sine-only DDSP.
- Loudness FiLM: Feature-wise Linear Modulation (FiLM) conditions the encoder on loudness, assisting the model in balancing the relative volumes of harmonics and noise.
- Post-Convolution Layer: A final learnable 1D convolution layer acts as a filter to suppress artifacts and balance the frequency response.
- Multi-Resolution Adversarial Loss: A GAN-based loss is incorporated to force the generation of fine spectral details, preventing the "muffled" output often associated with spectral-only losses.
Results and Discussion
The model was evaluated on the MNGU0 EMA dataset and a pseudo-labelled version of the LJ Speech dataset. The DDSP model consistently outperformed the HiFi-CAR baseline across multiple dimensions, particularly in efficiency.
Performance and Efficiency
The DDSP vocoder achieved a Word Error Rate (WER) of 6.67% and a Mean Opinion Score (MOS) of 3.74, surpassing the baseline by 1.63% and 0.16, respectively. Crucially, the model demonstrates extreme parameter efficiency. As shown in Figure 6, a version with only 0.4M parameters matches the quality of the 9M parameter SOTA model. Furthermore, the DDSP architecture is 4.9x faster during inference on a standard CPU, making it highly suitable for real-time applications.
Inductive Bias and GAN Loss
The success of the model stems from its strong inductive bias; by using oscillators and filters that inherently understand sound production, the neural network only needs to learn control signals. Additionally, while GAN loss slightly decreased objective similarity scores (like PESQ), it significantly improved subjective human preference (MOS) by reducing spectral over-smoothing.
Conclusion
This work presents a parameter-efficient and high-speed articulatory vocoder based on DDSP. By combining neural control with traditional DSP modules, the model achieves state-of-the-art quality with a fraction of the computational cost. Future research will focus on extending the model to multi-speaker scenarios and reducing dependency on high-quality pitch and loudness inputs for fully autonomous "silent speech" applications.
Summary updated