Enhancing Intelligibility of Dysarthric Speech Using Gated Convolutional-Based Voice Conversion System

Chen-Yu Chen, Wei-Zhong Zheng, Syu‐Siang Wang, Yu Tsao, Pei-Chun Li, Ying-Hui Lai2020 · DOI

TL;DR

This paper introduces a novel voice conversion (VC) system using a Gated Convolutional Neural Network (Gated CNN) to improve the intelligibility of dysarthric speech. The proposed system outperforms traditional CNN and BLSTM models in both objective and subjective evaluations, offering a more parameter-efficient solution for enhancing communication for individuals with speech impairments.

What Problem Does This Paper Address?

Dysarthria, a speech disorder resulting from neurological damage, significantly impairs speech intelligibility due to issues like phoneme loss, unstable prosody, and imprecise articulation. While Augmentative and Alternative Communication (AAC) devices exist, they can be inefficient. Voice conversion (VC) offers a promising avenue to enhance the intelligibility of dysarthric speech by transforming it into clearer speech while preserving linguistic content. Existing deep learning VC models, while effective, often require a large number of parameters, increasing hardware costs. This paper aims to address this by exploring a more efficient deep learning architecture, the Gated CNN, for dysarthric speech VC.

Methods

The core of this research is a voice conversion system built upon a Gated Convolutional Neural Network (Gated CNN). The system operates in two phases:

  • Training Phase: Parallel dysarthric speech ($s_j$) and target speech ($t_j$) are used. The dysarthric speech is processed through a feature extraction unit to obtain Phonetic Posteriorgrams (PPGs). Specifically, mel-cepstral coefficients (MCEPs) are extracted and fed into a speaker-dependent ASR model to generate 74-dimension PPGs. The target speech is converted into 80-dimension mel-spectrograms. These PPGs and mel-spectrograms serve as the input and output for training the Gated CNN model.
  • Conversion Phase: The trained Gated CNN model takes PPGs extracted from new dysarthric speech as input and converts them into mel-spectrograms. These mel-spectrograms are then fed into a WaveRNN vocoder to synthesize the final converted speech waveform.

The Gated CNN architecture is highlighted for its "gated linear unit" (GLU), which allows the network to control information flow through its layers. This is contrasted with standard CNNs and Bidirectional Long Short-Term Memory (BLSTM) networks, which were also evaluated for comparison.

Datasets

  • Training Data: 576 parallel dysarthric-normal utterances were used. These were sourced from the Taiwan Mandarin Hearing in Noise Test (MHINT) corpus.
  • Testing Data: 32 duplicate utterances and 32 outside utterances from the same dysarthric speaker were recorded for testing.
  • ASR Model Training: The speaker-dependent ASR (SD-ASR) system used for PPG extraction was trained on 3160 utterances from ten dysarthric speakers and one normal speaker using the Kaldi toolkit.

Key Insights and Ideas

  • Gated CNN's Superiority: The Gated CNN architecture demonstrably outperforms both standard CNN and BLSTM models in enhancing dysarthric speech intelligibility. This suggests its ability to better capture and process the complex features of dysarthric speech.
  • Parameter Efficiency: A significant advantage of the Gated CNN is its reduced parameter count compared to BLSTM models, making it a more practical choice for hardware implementation and potentially lowering system costs for users.
  • PPGs as Effective Features: The use of Phonetic Posteriorgrams (PPGs) as input features proved effective for the Gated CNN model in the VC task.
  • Spectrogram Detail: Visual analysis of spectrograms revealed that the Gated CNN produced more detailed and clearer formants, indicating potentially higher speech quality.

Key Results

  • Google ASR Evaluation:
    • Average Performance (Duplicate & Outside Tests): Gated CNN achieved the highest average recognition rate of 81.0%, significantly outperforming BLSTM (67.0%) and CNN (75.0%).
  • Duplicate Test: Gated CNN's best performance reached 85.9% accuracy.
  • Outside Test: Gated CNN's best performance reached 76.1% accuracy.
  • Word Correct Rate (WCR): The Gated CNN converted speech achieved an 87.8% accuracy rate, surpassing BLSTM (58.8%) and CNN (77.5%).
  • Mean Opinion Score (MOS) for Naturalness: The Gated CNN also scored highest in subjective naturalness evaluations, with an average MOS of 3.92, compared to BLSTM (2.6) and CNN (3.36).
  • Parameter Count: The best-performing Gated CNN model had approximately 30 million parameters, while the best BLSTM model had 85 million parameters.

Limitations and Future Work

  • Limitations: The paper acknowledges that while Gated CNN performs better, the sequential and hierarchical structures of speech, which BLSTM architectures can preserve, might not be as effectively captured by CNNs in general. However, the Gated CNN's gating mechanism appears to mitigate this to some extent.
  • Future Work: Not explicitly detailed in the paper, but the success of the Gated CNN suggests its potential for further exploration in other speech processing tasks and for different types of speech impairments. The efficiency of the Gated CNN also opens avenues for real-time applications on resource-constrained devices.

Summary updated

BibTeX
@misc{chen2020enhancing,
  author = {Chen-Yu Chen and Wei-Zhong Zheng and Syu‐Siang Wang and Yu Tsao and Pei-Chun Li and Ying-Hui Lai},
  title = {{Enhancing Intelligibility of Dysarthric Speech Using Gated Convolutional-Based Voice Conversion System}},
  year = {2020},
  doi = {10.21437/interspeech.2020-1367},
}
Figure