TL;DR
This paper introduces a novel approach to separating unvoiced speech (like "s" or "f" sounds) from background noise, a task that's been surprisingly overlooked. By first isolating and removing voiced speech and periodic interference, the system then uses spectral subtraction and clever grouping strategies to effectively extract unvoiced speech, significantly outperforming previous methods.
What Problem Does This Paper Address?
Extracting speech from noisy environments is a long-standing challenge in audio processing. While significant progress has been made in separating voiced speech (sounds with clear pitch, like vowels), unvoiced speech – sounds characterized by noise rather than pitch – has received much less attention. These unvoiced sounds are crucial for intelligibility, but their weak energy and lack of periodicity make them particularly susceptible to interference, often getting lost or masked by background noise. This paper aims to bridge this gap by developing a robust method for unvoiced speech segregation.
Methods
The proposed system builds upon Computational Auditory Scene Analysis (CASA) principles and introduces a two-stage approach:
- Voiced Speech Segregation and Periodic Interference Removal: The system first employs a sophisticated tandem algorithm to segregate voiced speech. Crucially, it then removes not only the identified voiced speech but also any periodic components from the interference. This step is vital because unvoiced speech lacks periodicity, so removing periodic interference simplifies the remaining mixture. Periodic signal removal is achieved by identifying time-frequency (T-F) units with high cross-channel correlation, indicating periodicity. Thresholds for this removal are carefully tuned to minimize the loss of actual unvoiced speech.
- Unvoiced Speech Segmentation and Grouping: After periodic interference is removed, the remaining mixture primarily consists of unvoiced speech and aperiodic noise.
- Spectral Subtraction: The system estimates the noise energy in unvoiced T-F intervals by averaging the energy of masked units (identified as interference) in neighboring voiced intervals. This estimated noise is then used in a spectral subtraction process to generate T-F segments that are likely to contain unvoiced speech. An over-subtraction factor of 2 is found to be beneficial.
- Grouping: The segments identified by spectral subtraction might still contain residual noise. To refine this, two grouping strategies are proposed:
- Thresholding: Segments are classified as unvoiced speech if their energy distribution is concentrated in higher frequencies (lower bound > 2 kHz or upper bound > 6 kHz), a characteristic of unvoiced speech but not typical noise residues.
- Bayesian Classification: A more formal approach uses a Multilayer Perceptron (MLP) trained on features derived from segment frequency bounds and size to classify segments as either unvoiced speech or interference.
Datasets
- Training:
- TIMIT database (training portion) mixed with 100 nonspeech interferences at 0 dB SNR for training the voiced speech segregation MLP.
- IEEE sentence database mixed with 15 nonspeech interferences for tuning periodic signal removal thresholds.
- 0-dB mixtures of 100 speech utterances and 15 interferences for training the grouping MLP.
- Evaluation:
- IEEE sentence database mixed with 15 nonspeech interferences (electric fan, white noise, crowd noise, rain, babble noise, rock music, wind, cocktail party noise, clock alarm, traffic noise, siren, bird chirp, telephone ring) at various SNR levels (-5, 0, 5, 10, 15 dB).
- Bird chirp noise at 0 dB SNR for evaluating noise energy estimation accuracy.
Key Insights and Ideas
- The Importance of Periodic Signal Removal: Removing periodic components (both voiced speech and periodic interference) before tackling unvoiced speech significantly cleans up the signal and improves the accuracy of subsequent noise estimation. This is a key differentiator from prior work.
- Acoustic-Phonetic Properties for Grouping: Leveraging the distinct frequency characteristics of unvoiced speech (higher energy in middle and high frequencies) provides a simple yet effective way to distinguish it from residual noise.
- Simplicity vs. Performance: The proposed thresholding method for grouping is computationally simpler than complex classification or noise-type dependent methods, yet achieves comparable or better results.
- Low SNR Performance is Crucial: The paper highlights that significant gains at low input SNRs are particularly beneficial for individuals with hearing loss, making this research highly relevant for assistive technologies.
Key Results
- The proposed system achieves an average of 1.6 dB SNR improvement over the previous state-of-the-art unvoiced speech segregation system by Hu and Wang [1] across various input SNRs.
- The system significantly outperforms spectral subtraction alone, with a gap of up to 9.3 dB SNR gain at -5 dB input SNR.
- Periodic signal removal leads to a uniformly smaller RMS error in noise energy estimation compared to not performing this step, especially at higher frequencies.
- The grouping method using thresholding is computationally much simpler than MLP-based classification and yields similar performance, suggesting the problem is well-posed by the frequency characteristics.
Limitations and Future Work
- Computational Complexity: While the proposed method is more efficient than prior work, it still involves spectral subtraction, which can be computationally intensive.
- Generalizability of Grouping: The paper notes that while thresholding and Bayesian classification performed similarly, further exploration into incorporating acoustic-phonetic features could potentially yield even better results.
- Real-world Noise Complexity: While a diverse set of interferences was used, real-world scenarios can involve more complex and dynamic noise mixtures.
- Future Work: The authors suggest exploring more advanced grouping strategies and potentially integrating their unvoiced speech segregation with existing voiced speech separation systems for a more comprehensive solution.
Summary updated