TL;DR
This paper presents a novel two-step signal processing approach to improve speech intelligibility in noisy environments, specifically for synthetic speech generated by harmonic models. By subtly adjusting spectral tilt and dynamically compressing audio energy, the system significantly boosts comprehension without altering the overall signal energy, offering a cost-effective enhancement for speech synthesis engines.
What Problem Does This Paper Address?
Speech synthesizers are typically trained on clean speech, meaning their output can become difficult to understand when played in noisy conditions. While existing methods involve re-recording or complex voice conversion, this paper focuses on enhancing existing synthetic speech after generation. The challenge is to make synthetic speech more robust to noise without requiring extensive retraining or introducing noticeable artifacts, particularly within the framework of harmonic model-based speech synthesis.
Methods
The proposed system operates on the parameters of a harmonic model (HM), which represents speech as a sum of harmonically related sinusoids. The enhancement is achieved in two sequential steps:
- Spectral Slope Modification: This step aims to mimic the effect of increased vocal effort, which is known to improve intelligibility in noise. It involves increasing the spectral tilt (the rate at which higher frequencies decrease in amplitude) by a configurable amount. This is achieved by modifying the harmonic amplitudes based on their order and applying a normalization factor to preserve the frame's overall energy.
- Dynamic Range Compression (DRC): This step addresses the masking of low-energy speech components (like plosives and fricatives) by noise. It redistributes the signal's energy by amplifying quieter segments while leaving louder segments relatively untouched. The authors implement a nonlinear mapping function on the local energy of the signal, effectively boosting the energy of weak phonemes. The overall signal energy is then re-normalized.
Crucially, both steps operate on the harmonic amplitudes, making them easily integrable into HM-based waveform generation modules without significant computational overhead.
Datasets
- Database: 240 phonetically balanced sentences from the Harvard corpus.
- Noise: Speech-shaped noise was used, with the noise spectrum matched to the average human speech spectrum for a more realistic evaluation.
- SNR Levels: Experiments were conducted at Signal-to-Noise Ratios (SNR) of -9dB, -4dB, and 1dB.
The paper also references data from a previous listening test [24] for validation purposes.
Key Insights and Ideas
- Spectral Tilt as an Intelligibility Cue: The paper highlights that increasing spectral tilt, mimicking higher vocal effort, can improve intelligibility. This is a well-understood phenomenon in speech production but is now being leveraged as a signal processing technique.
- DRC for Low-Energy Phonemes: Dynamic range compression is shown to be effective in boosting the energy of crucial low-energy consonants that are often masked by noise, thereby improving overall comprehension.
- Harmonic Model Advantage: Operating directly on harmonic model parameters is computationally efficient and allows for seamless integration into existing statistical parametric speech synthesis systems.
- Objective-Subjective Correlation: The study demonstrates a strong correlation (0.8) between their implemented Speech Intelligibility Index (SII) and subjective listening test results, validating SII as a reliable objective measure for this enhancement.
Key Results
- Objective Improvement: The proposed system, using both spectral slope modification and DRC, yielded an approximate 0.1-point increment in the Speech Intelligibility Index (SII) score compared to the original natural speech.
- Subjective Improvement: A perceptual test showed that the enhanced signals significantly increased the number of correctly understood words, particularly in low SNR conditions (e.g., a four-fold increase).
- Parameter Sensitivity: The optimal spectral slope parameter (m) was found to range from 15 to 30 dB/decade for negative SNRs. The choice of m had less impact within the 20-25 dB/decade range.
Limitations and Future Work
- Parameter Optimization: While the paper suggests an automatic method for determining the optimal spectral slope parameter (m) based on SII, it acknowledges that this is not straightforward and requires considering perceptual quality.
- Noise Types: The current evaluation focused on speech-shaped noise. Experiments involving other noise types (e.g., cafeteria, car, high-frequency noise) are in progress.
- Phase Modifications: The current system only manipulates harmonic amplitudes. Future work aims to explore the impact of phase modifications on intelligibility.
- Generalizability: While informal tests suggest good performance across different voices and languages, further systematic evaluation is implied.
Summary updated