A fast algorithm for improved intelligibility of speech-in-noise based on frequency and time domain energy reallocation

Tudor-Cătălin Zorilă, Yannis StylianouInterspeech 2015

TL;DR

This paper introduces a novel, computationally efficient algorithm (fSER) for enhancing speech intelligibility in noisy environments. It achieves comparable intelligibility gains to existing state-of-the-art methods but with a remarkable 90% reduction in computational load, making it highly suitable for real-time applications.

What Problem Does This Paper Address?

The core challenge addressed is improving the intelligibility of speech when it's mixed with background noise. This is a critical problem in many real-world scenarios, from noisy train stations to crowded events. Traditional methods of simply increasing volume can be detrimental to hearing and may not effectively improve comprehension. The paper builds upon research into how humans naturally adapt their speech in noisy conditions (like the Lombard reflex) and recent challenges (like the Hurricane Challenges) that specifically sought algorithms to enhance speech intelligibility without increasing the overall signal power. Existing successful algorithms, while effective, are often computationally expensive, hindering their deployment in real-time systems.

Methods

The proposed algorithm, fSER (fast Spectral Energy Reallocation), is inspired by previous work but significantly optimized for speed. It operates in the frequency domain and employs a three-stage process:

  1. Preservation of Low Frequencies (PLF): A filter is used to maintain the spectral energy in the low-frequency range (below 50 Hz and above 400 Hz).
  2. High-Frequency Boost: A pre-emphasis filter is applied to boost the signal-to-noise ratio (SNR) in higher frequencies.
  3. Spectral Contrast Enhancement (fSCE): This is a novel contribution. It's a frequency-domain method designed to increase the contrast between spectral peaks and valleys in the mid-frequencies (250 Hz to 4500 Hz). Unlike previous, computationally intensive methods, fSCE uses a faster approach involving spectral tilt estimation and a static input/output contrast characteristic (IOCC).

The outputs of these stages are mixed with specific weights, and the phase spectrum is preserved to ensure a natural-sounding re-synthesis. Additionally, the paper explores combining fSER with a Dynamic Range Compression (DRC) stage (fSER+DRC), similar to successful prior methods, to further amplify low-energy speech segments. A key constraint maintained throughout is that the global signal RMS (root mean square) power remains the same before and after enhancement.

Datasets

The evaluation used the first 180 Harvard sentences, spoken by a male native British English talker. These clean speech samples were mixed with two types of noise: Speech Shaped Noise (SSN) and Competing Speaker (CS) noise. The mixtures were created at various Signal-to-Noise Ratios (SNRs):

  • SSN: -9 dB (Low SNR), -4 dB (Mid SNR), 1 dB (High SNR)
  • CS: -21 dB (Low SNR), -14 dB (Mid SNR), -7 dB (High SNR)

The paper does not introduce new datasets but utilizes existing, well-established speech and noise corpora for evaluation.

Key Insights and Ideas

  • Computational Efficiency is Key for Real-Time: The primary insight is that achieving comparable intelligibility gains with a 90% reduction in computational load (fSER vs. tSER) is a significant breakthrough for real-time applications.
  • Frequency Domain Processing Can Be Fast: By carefully designing frequency-domain operations and preserving phase information, the authors demonstrate that complex speech enhancement can be achieved efficiently.
  • Spectral Contrast Matters: The novel fSCE method highlights the importance of enhancing the contrast between spectral peaks and valleys in the mid-frequencies for intelligibility.
  • DRC Still Adds Value: Combining the fSER algorithm with a dynamic range compression stage (fSER+DRC) consistently improves intelligibility further, suggesting that time-domain energy reallocation remains a valuable component.

Key Results

  • Intelligibility Gains: The fSER algorithm achieved intelligibility gains comparable to existing state-of-the-art methods like tSER and SSDRC across various noise conditions and SNRs, as measured by keyword recognition rates.
  • Computational Speedup: fSER demonstrated a remarkable 90% lower computational load compared to tSER, with average run times significantly lower (e.g., fSER around 0.2s vs. tSER around 3.3s for a set of sentences).
  • fSER+DRC Performance: The combined fSER+DRC approach showed consistent improvements over fSER alone, further boosting intelligibility.
  • Statistical Significance: Statistical tests (Fisher's Least Significant Differences) confirmed that fSER performed similarly to tSER in terms of intelligibility, while being much faster.

Limitations and Future Work

  • Real-time Implementation: While the algorithm is designed for real-time, the paper states that as future work, they intend to implement the indicated approach in real-time.
  • Audible Distortions: Although the authors aimed for a good trade-off, the paper acknowledges that the IOCC curve was empirically derived to balance intelligibility gains and audible distortions. Further fine-tuning might be explored.
  • Limited Noise Types: The evaluation focused on Speech Shaped Noise and Competing Speaker noise. Testing with a wider variety of real-world noise types could provide a more comprehensive understanding of the algorithm's robustness.

Summary updated

BibTeX
@inproceedings{zorila2015fast,
  author = {Tudor-Cătălin Zorilă and Yannis Stylianou},
  title = {{A fast algorithm for improved intelligibility of speech-in-noise based on frequency and time domain energy reallocation}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure