Abstract: We propose a new paradigm for maintaining speaker identity in dysarthric voice conversion (DVC). The poor quality of dysarthric speech can be greatly improved by statistical VC, but as the normal speech utterances of a dysarthria patient are nearly impossible to collect, previous work failed to recover the individuality of the patient. In light of this, we suggest a novel, two-stage approach for DVC, which is highly flexible in that no normal speech of the patient is required. First, a powerful parallel sequence-to-sequence model converts the input dysarthric speech into a normal speech of a reference speaker as an intermediate product, and a nonparallel, frame-wise VC model realized with a variational autoencoder then converts the speaker identity of the reference speech back to that of the patient while assumed to be capable of preserving the enhanced quality. We investigate several design options. Experimental evaluation results demonstrate the potential of our approach to improving the quality of the dysarthric speech while maintaining the speaker identity.
1. Introduction
Dysarthria is a motor speech disorder resulting from neurological disruptions such as cerebral palsy or ALS. It often results in speech that is unintelligible due to phoneme loss and unstable prosody. While Voice Conversion (VC) can significantly improve speech clarity, it faces a major hurdle: identity preservation.
Most high-quality VC models require parallel data—recordings of the source and target saying the exact same sentences. To make a patient sound like a "normal" version of themselves, one would ideally need recordings of them speaking clearly. However, for many patients, such samples do not exist. This paper proposes a way to provide both high intelligibility and the patient's original "individuality" without needing any healthy samples from the patient.
2. Proposed Two-Stage Paradigm
The core of this research is a two-stage paradigm that utilizes a healthy third-party reference speaker as a bridge to solve the data scarcity problem.
As illustrated in Figure 1, the system operates in two distinct phases:
- Stage 1: Sequence-to-Sequence (Seq2Seq) Conversion: The input dysarthric speech is converted into the speech of a healthy reference speaker using a Transformer-based Voice Transformer Network (VTN). This stage handles timing and prosody globally, significantly boosting intelligibility and naturalness, though the voice identity is temporarily changed to that of the reference speaker.
- Stage 2: Non-parallel Frame-wise Restoration: The output from Stage 1 is fed into a Variational Autoencoder (VAE). This model is trained to change only the "time-invariant" characteristics (speaker identity) while leaving the "time-variant" characteristics (the now-clear pronunciation) intact. This effectively "restores" the patient's identity to the clear speech produced in Stage 1.
The system concludes with a Parallel WaveGAN (PWG) vocoder to synthesize the final high-fidelity waveform.
3. Experimental Evaluation
The researchers used a Mandarin corpus from a female patient with cerebral palsy and a reference dataset (TMSV) consisting of 17 healthy speakers. Models were pretrained on the Sinica COSPRO multi-speaker dataset to ensure a deep understanding of Mandarin phonetics. Objective intelligibility was measured using Syllable Error Rate (SER) via an ASR engine trained on AISHELL-1.
4. Key Results and Insights
The experimental results demonstrate a significant breakthrough in balancing clarity with identity.
- Intelligibility Gains: As shown in Figure 3, the original dysarthric speech had a baseline SER of 94.0%. After the two-stage conversion (using speaker SP09 as a reference), the SER dropped to 75.8%, an 18.2-point improvement.
- Identity Restoration: This was the study's primary success. While Stage 1 output (VTN only) had nearly 0% similarity to the patient, the Stage 2 output (VTN+VAE) reached nearly 50% similarity, proving the second stage successfully restores the patient's persona.
- Naturalness: Subjective naturalness scores (MOS) improved from a baseline of 2.37 to a peak of 2.65 (using reference SP07).
5. Conclusion and Future Work
While the two-stage approach successfully preserves identity without patient-specific "normal" data, the authors noted an "intelligibility leakage" where the VAE stage slightly re-introduces ambiguity. Future work will focus on improving factorization using hierarchical VAE structures or Phonetic Posteriorgrams (PPGs) to better isolate linguistic content from speaker identity, and automating the selection of the most compatible reference speaker using x-vectors.
Summary updated