TL;DR
This paper tackles the challenge of devoicing errors in speech from individuals with cleft lip and palate (CLP), a common issue that significantly impacts intelligibility. They propose a novel method to modify these errors by segmenting specific acoustic events (closure and voice onset time) and applying learned transformations to restore voicing characteristics, leading to improved speech clarity.
What Problem Does This Paper Address?
Individuals with cleft lip and palate (CLP) often experience speech disorders due to structural and functional differences in their vocal tract. One prevalent issue is the "devoicing error," where voiced stop consonants (like /b/, /d/, /g/) are produced as their voiceless counterparts (/p/, /t/, /k/). This manifests acoustically as a lack of the "voicebar" (a low-frequency component indicating vocal fold vibration) during the closure phase of the consonant and an extended voice onset time (VOT). These acoustic deviations significantly degrade speech intelligibility, making communication difficult. While speech therapy aims to correct these issues, this paper proposes an automated signal processing approach to enhance speech intelligibility by directly modifying these specific acoustic errors.
Methods
The core of this work lies in a two-stage signal processing approach:
- Segmentation: The first crucial step is to accurately identify the relevant acoustic regions within the speech signal that correspond to the devoicing error. This involves:
- Glottal Activity Detection (GAD): Using a Zero Frequency Filtering (ZFF) approach to pinpoint vocal fold closure instants.
- Burst Detection: Employing a Plosion Index (PI) metric to locate the sudden release of air pressure that marks the burst of a stop consonant.
- Region Specification: Based on the detected glottal activity and burst points, the algorithm defines the "closure interval" (where the voicebar should be) and the "voice onset time" (VOT) region. The paper notes that the closure interval is defined differently for word-initial (60ms before burst onset) and word-medial positions.
- Transformation: Once the regions are segmented, specific transformations are applied to modify the acoustic characteristics:
- Voicebar Modification: A transformation matrix is learned for the closure interval using Non-negative Matrix Factorization (NMF). This matrix is derived by comparing acoustic features from normal voiced stops and devoiced CLP stops.
- VOT Modification: If the VOT of the devoiced stop deviates beyond a certain threshold, it is also modified. The paper mentions aligning the warped version of the CLP speaker's VOT with that of a normal speaker's VOT using Dynamic Time Warping (DTW).
- NMF for Transformation Matrix Learning: The paper details using NMF to learn transformation matrices ($\hat{W}$) from source (devoiced CLP) and target (normal voiced) spectral dictionaries. This is achieved by minimizing the Kullback-Leibler divergence between the target and transformed source spectra. The enhanced speech is then reconstructed by combining the modified segments with the unprocessed parts of the utterance.
Datasets
The speech data used in this work was collected from the All India Institute of Speech and Hearing (AIISH), Mysore, India. It comprises:
- 60 native Kannada speakers.
- 29 speakers with Cleft Lip and Palate (CLP) (9 ± 2 years old).
- 31 non-CLP control speakers (10 ± 2 years old). The database includes vowel phonation, nonsense VCV, and CVCV words, and meaningful words and sentences. For this specific study, only nonsense CVCV words were used for detection and enhancement. The speech samples were recorded at 48 kHz sampling rate and 16-bit resolution, then down-sampled to 16 kHz for processing.
Key Insights and Ideas
- Decomposition of Devoicing: The paper's strength lies in recognizing that devoicing isn't a single acoustic anomaly but a combination of issues in specific temporal and spectral regions (closure and VOT). This allows for targeted modifications.
- NMF for Adaptive Transformation: Using NMF to learn transformation matrices is a powerful technique that allows the model to adapt to the specific spectral characteristics of both normal and disordered speech, leading to more nuanced modifications.
- Importance of Segmentation: Accurate segmentation of acoustic events (burst, glottal activity) is presented as a critical prerequisite for effective transformation, highlighting the interplay between feature extraction and modification.
- Subjective Evaluation Matters: The inclusion of a subjective intelligibility test, where listeners rate the perceived voicing, provides crucial validation beyond objective acoustic measures.
Key Results
- Improved Acoustic Characteristics: Visual inspection of spectrograms (Figure 5) shows that the modified speech exhibits a clearer voicebar in the closure interval and enhanced VOT characteristics, making them acoustically more similar to normal voiced stops.
- Subjective Intelligibility Improvement: A perceptual evaluation test with ten listeners showed significant improvement in perceived voicing. The transformed speech samples received scores tending towards 100 (representing voiced stops), compared to the original deviant CLP speech.
- High Identification Accuracy: For the subjective test, the transformed speech samples were correctly identified as voiced stops with an average accuracy of 72.1% (ranging from 60% to 85% across listeners), a substantial improvement over the original deviant speech which averaged 8.5% (ranging from 0% to 20%).
- Segmentation Accuracy: The segmentation algorithm achieved high accuracy for burst detection, with 96% for /b/ → /p/, 84.09% for /d/ → /t/, and 73.91% for /g/ → /k/.
Limitations and Future Work
- Segmentation Accuracy Variability: The authors acknowledge that in real-world environments, segmentation accuracy might vary, potentially affecting the enhancement outcome.
- Phoneme-Specific Enhancement: The current work focuses on phoneme-specific sound units in consonant-vowel-consonant-vowel structures.
- Future Directions: The authors suggest further exploration to address challenges like reverberation and background noise, and to extend the approach to meaningful words and spontaneous speech to meet real-world scenarios. They also propose studying the effect of these factors on the proposed enhancement method.
Summary updated