TL;DR
This paper tackles the challenge of creating personalized synthetic voices for individuals with degenerative speech disorders, like MND, who may have already lost speech clarity. They propose using HMM-based speech synthesis with novel "voice reconstruction" techniques to repair disordered speech while preserving the speaker's unique vocal identity, showing promising results in improving intelligibility.
What Problem Does This Paper Address?
Losing the ability to speak due to degenerative diseases like Motor Neuron Disease (MND) or Parkinson's is devastating, impacting not just communication but also personal identity. While "voice banking" (recording speech before it's lost) and text-to-speech (TTS) synthesizers exist, they have limitations. Voice banking requires clear speech, which isn't always possible if deterioration is rapid. Current TTS systems, while intelligible, often lack personalization, failing to capture the individual's vocal identity. This paper addresses the critical need for personalized synthetic voices that can be created even from disordered speech, aiming to restore both communication and a sense of self for affected individuals.
Methods
The core of this work lies in leveraging Hidden Markov Model (HMM)-based speech synthesis (HTS) and introducing "voice reconstruction" techniques. HTS is chosen for its ability to create personalized voices with limited data through speaker adaptation. The key innovation is the ability to repair synthetic speech generated from disordered recordings. This is achieved by strategically substituting components of the HTS acoustic models.
The authors explore three substitution strategies:
- Baseline Model Substitution: Replaces duration and aperiodicity models, and global variances. This helps with issues like slow speech rate and hoarseness but doesn't fix articulation problems.
- Component-wise Model Substitution: Allows for more granular substitution of specific acoustic parameters (like mel-cepstrum coefficients and log-F0 dynamics) that are less correlated with speaker identity. This aims to reduce articulation disorders without altering timbre.
- Context-dependent Model Substitution: An extension of the component-wise approach that uses contextual information (phoneme types, diphthongs) to decide when to substitute mel-cepstrum coefficients entirely, specifically targeting intelligibility issues in certain speech sounds.
Datasets
- Training Data: The authors trained "average voice models" using a database of approximately 300 healthy voice donors with diverse accents. Each speaker contributed about an hour of speech (400 sentences).
- Patient Data: Recordings were made from a single MND patient.
- A 1-hour recording was made shortly after diagnosis when the speech was considered "healthy."
- A 15-minute recording was made 10 months later, exhibiting speech disorders typical of MND (hoarseness, breathiness, fluency disruption, reduced articulation, monotonic prosody).
- Evaluation Data: Semantically unpredictable sentences were used for intelligibility tests. A specific sentence was used for speaker similarity tests.
Key Insights and Ideas
- Reconstruction is Possible: The paper demonstrates that it's feasible to reconstruct a synthetic voice from disordered speech, not just clone a healthy voice.
- Preserving Identity is Key: The goal is to repair intelligibility without erasing the speaker's unique vocal characteristics.
- Context Matters for Repair: The context-dependent substitution strategy shows superior results, highlighting that specific speech sounds and contexts require tailored repair approaches.
- Disordered Speech Has Structure: The authors note that articulatory errors in disordered speech are often consistent and predictable, making model-based repair strategies effective.
- Synthetic Reference is Valid: Using a synthetic "healthy" voice clone as a reference for subjective evaluation proved to be a valid approach, avoiding biases from natural recording quality loss.
Key Results
- Intelligibility Improvement: The context-dependent reconstruction strategy (IR_v2) significantly reduced the Word Error Rate (WER) compared to a direct clone of the disordered speech (IC). While the baseline strategy (IR_v0) surprisingly increased WER, the component-wise (IR_v1) and context-dependent (IR_v2) strategies showed improvements.
- Speaker Similarity: The reconstructed voices (IR_v1 and IR_v2) achieved higher similarity scores to the reference "healthy" voice clone (HC) than the average voice model (AV) or the direct clone of disordered speech (IC). The context-dependent strategy (IR_v2) showed a notable improvement in similarity.
- Subjective Evaluation: A listening test with native English speakers indicated that the reconstructed voices were perceived as more similar to the target vocal identity and more intelligible than the direct clone of disordered speech.
Limitations and Future Work
- Single Patient Study: The evaluation was conducted on a single MND patient. The authors acknowledge that results may vary across different individuals and types of speech disorders.
- Generalizability: While the principles are believed to be generalizable to other degenerative or acquired speech disorders, further validation with a wider range of patients is needed.
- Future Work: The authors suggest that these reconstruction strategies could be generalized to other speech disorders. Further research could explore more sophisticated context-dependent rules and potentially integrate other signal processing techniques for even finer-grained repair.
Summary updated