Fast, High-Quality and Parameter-Efficient Articulatory Synthesis Using Differentiable DSP

Yisi Liu, Bohan Yu, Drake Lin, Peter Wu, Cheol Jun Cho, Gopala Krishna Anumanchipalli

Abstract

Articulatory trajectories like electromagnetic articulography (EMA) provide a low-dimensional representation of the vocal tract filter and have been used as natural, grounded features for speech synthesis. Differentiable digital signal processing (DDSP) is a parameter-efficient framework for audio synthesis. Therefore, integrating low-dimensional EMA features with DDSP can significantly enhance the computational efficiency of speech synthesis. In this paper, we propose a fast, high-quality, and parameter-efficient DDSP articulatory vocoder that can synthesize speech from EMA, F0, and loudness. We incorporate several techniques to solve the harmonics / noise imbalance problem, and add a multiresolution adversarial loss for better synthesis quality. Our model achieves a transcription word error rate (WER) of 6.67% and a mean opinion score (MOS) of 3.74, with an improvement of 1.63% and 0.16 compared to the state-of-the-art (SOTA) baseline. Our DDSP vocoder is 4.9 x faster than the baseline on CPU during inference, and can generate speech of comparable quality with only 0.4 M parameters, in contrast to the 9 M parameters required by the SOTA.
BibTeX
@inproceedings{liu2024fast,
  author = {Yisi Liu and Bohan Yu and Drake Lin and Peter Wu and Cheol Jun Cho and Gopala Krishna Anumanchipalli},
  title = {{Fast, High-Quality and Parameter-Efficient Articulatory Synthesis Using Differentiable DSP}},
  booktitle = {SLT},
  year = {2024},
  doi = {10.1109/slt61566.2024.10832354},
}

Loading related content...

Figure