Fast, High-Quality and Parameter-Efficient Articulatory Synthesis Using Differentiable DSP

Yisi Liu, Bohan Yu, Drake Lin, Peter Wu, Cheol Jun Cho, Gopala Krishna Anumanchipalli

TL;DR

Researchers from UC Berkeley have developed a new articulatory vocoder that synthesizes high-quality speech from physical vocal tract movements (EMA) using Differentiable Digital Signal Processing (DDSP). Their model is not only more accurate and natural-sounding than current state-of-the-art neural vocoders, but it is also 4.9x faster on a standard CPU and can achieve comparable quality with 22x fewer parameters (0.4M vs. 9M).

What Problem Does This Paper Address?

Articulatory synthesis is the process of generating speech from the physical movements of human articulators (like the tongue, lips, and jaw), typically measured via electromagnetic articulography (EMA). This is a powerful tool for medical applications—such as helping patients with vocal cord disorders or decoding brain signals into speech—because it is more interpretable and controllable than standard text-to-speech.

However, existing state-of-the-art (SOTA) models, like HiFi-CAR, are computationally expensive and parameter-heavy. This makes them difficult to deploy on "edge devices" (like smartphones or medical implants) where memory and processing power are limited. The challenge is to create a model that maintains high audio fidelity while being small enough and fast enough for real-time, low-footprint applications.

Methods

The authors utilize a Differentiable Digital Signal Processing (DDSP) framework. Unlike pure neural vocoders that try to learn how to generate raw waveforms from scratch, DDSP combines a neural network "encoder" with traditional, well-understood DSP modules.

Key technical innovations include:

  • Harmonic-plus-Noise (H+N) Model: The system splits speech into two parts: periodic harmonics (voiced sounds) and filtered noise (unvoiced sounds like "s" or "f").
  • Enhanced Expressivity: While standard DDSP often uses only sine waves, this model uses both sine and cosine harmonics to better approximate a Fourier series, allowing for richer tonal representation.
  • Loudness FiLM: They use Feature-wise Linear Modulation (FiLM) to condition the encoder on loudness, helping the model balance the volume of harmonics versus noise.
  • Post-Convolution Layer: A learnable 1D convolution layer is added at the end to act as a final filter, suppressing artifacts and balancing the overall frequency response.
  • Multi-Resolution Adversarial Loss: To prevent the "muffled" sound common in models trained only on spectral distance, they added a GAN-based loss that forces the model to generate finer, more realistic spectral details.

Datasets

The study utilized two primary datasets:

  1. MNGU0 EMA Dataset: A specialized dataset containing 75 minutes of 16 kHz male speech synchronized with 12-dimensional EMA data (tracking the x and y coordinates of the jaw, lips, and tongue).
  2. LJ Speech Pseudo-Labelled Dataset: To test the model on a larger scale, they used the 24-hour LJ Speech dataset. Since it lacks physical EMA data, they used an Acoustic-to-Articulatory Inversion (AAI) model to generate "pseudo-EMA" labels, demonstrating that their vocoder works even with estimated physical features.

Key Insights and Ideas

  • Inductive Bias is a Superpower: By using DSP modules (oscillators and filters) that already "know" how sound works, the neural network only has to learn the control signals (like pitch and filter coefficients) rather than every individual sample of a waveform. This is why the model can be so much smaller than its competitors.
  • The GAN Trade-off: The ablation study revealed a classic machine learning tension: training with GAN loss actually decreased objective similarity scores (like PESQ) but significantly increased subjective human preference scores (MOS). This confirms that "mathematically perfect" reconstruction often sounds "perceptually over-smoothed" to human ears.
  • Balancing Act: One of the hardest parts of DDSP is preventing the noise component from overpowering the harmonics. The authors found that the combination of the attenuation hyperparameter ($\gamma$) and the post-convolution layer was essential to prevent the model from "giving up" on harmonics and just producing shaped noise.

Key Results

The DDSP model outperformed the SOTA HiFi-CAR baseline across nearly every metric:

  • Intelligibility: Achieved a Word Error Rate (WER) of 6.67%, a 1.63% improvement over the baseline.
  • Quality: Earned a Mean Opinion Score (MOS) of 3.74, beating the baseline by 0.16.
  • Speed: It is 4.9x faster than the baseline on an Apple M1 CPU, making it highly viable for real-time use.
  • Efficiency: A tiny 0.4M parameter version of the model matched the quality of the 9M parameter SOTA model. Even their "full" 9M parameter version was significantly faster due to the efficiency of the DSP operations.

Limitations and Future Work

The authors acknowledge a few areas for growth:

  • Single Speaker: The current experiments were conducted on single-speaker datasets. Expanding the model to handle multi-speaker capabilities is a primary goal for future research.
  • Feature Dependency: The model relies on high-quality F0 (pitch) and loudness inputs. In a real-world "silent speech" or brain-to-speech application, these features would also need to be predicted accurately from the source signal.
  • Articulatory Precision: While the model is fast and high-quality, the paper doesn't deeply explore how sensitive the synthesis is to very small errors in the EMA trajectories, which might be common in clinical settings.

Summary updated

Abstract

Articulatory trajectories like electromagnetic articulography (EMA) provide a low-dimensional representation of the vocal tract filter and have been used as natural, grounded features for speech synthesis. Differentiable digital signal processing (DDSP) is a parameter-efficient framework for audio synthesis. Therefore, integrating low-dimensional EMA features with DDSP can significantly enhance the computational efficiency of speech synthesis. In this paper, we propose a fast, high-quality, and parameter-efficient DDSP articulatory vocoder that can synthesize speech from EMA, F0, and loudness. We incorporate several techniques to solve the harmonics / noise imbalance problem, and add a multi-resolution adversarial loss for better synthesis quality. Our model achieves a transcription word error rate (WER) of 6.67% and a mean opinion score (MOS) of 3.74, with an improvement of 1.63% and 0.16 compared to the state-of-the-art (SOTA) baseline. Our DDSP vocoder is 4.9x faster than the baseline on CPU during inference, and can generate speech of comparable quality with only 0.4M parameters, in contrast to the 9M parameters required by the SOTA.
BibTeX
@inproceedings{liu2024fast,
  author = {Yisi Liu and Bohan Yu and Drake Lin and Peter Wu and Cheol Jun Cho and Gopala Krishna Anumanchipalli},
  title = {{Fast, High-Quality and Parameter-Efficient Articulatory Synthesis Using Differentiable DSP}},
  booktitle = {SLT},
  year = {2024},
  doi = {10.1109/slt61566.2024.10832354},
}

Loading related content...

Figure