Interfacing Sound Stream Segregation to Automatic Speech Recognition — Preliminary Results on Listening to Several Sounds Simultaneously

AAAI 1996

TL;DR

This paper explores how to improve automatic speech recognition (ASR) in noisy environments by first segregating individual speech streams from a mixture of sounds. They propose a novel method that combines harmonic sound analysis with binaural cues and residue substitution, then interfaces this with HMM-based ASR, achieving a 75% cumulative accuracy for recognizing words in a two-speaker scenario.

What Problem Does This Paper Address?

The "cocktail party effect" describes our remarkable ability to focus on a single voice in a noisy, multi-talker environment. While psychoacoustics has studied this for decades, computational auditory scene analysis (CASA) is a newer field aiming to replicate this with computers. A key application of CASA is as a front-end for Automatic Speech Recognition (ASR) systems, which currently struggle significantly in noisy or multi-speaker conditions. Existing ASR systems often treat everything other than the target speech as noise, but this paper proposes a more sophisticated approach: actively segregating the target speech stream from other sounds before feeding it to the ASR. The challenge lies in how to effectively segregate speech streams and then interface this segregated signal with ASR systems without introducing further distortions that degrade recognition performance.

Methods

The core of the proposed system is a binaural harmonic-based stream segregation (Bi-HBSS) system. This system builds upon previous work by:

  • Harmonic Stream Fragment Extraction: It identifies and extracts fragments of harmonic structures (like vowels) from the sound mixture.
  • Binaural Cue Integration: Crucially, it uses binaural input (left and right ear signals) to calculate interaural time differences (ITD) and interaural intensity differences (IID). This directional information helps refine the segregation process and group fragments originating from the same source.
  • Harmonic Grouping: Fragments are grouped into streams based on proximity of fundamental frequencies (F-grouping) or direction (D-grouping). The paper found D-grouping to be most effective.
  • Residue Substitution: Non-harmonic components (like consonants) are implicitly handled by substituting the "residue" – the parts of the sound mixture not accounted for by the extracted harmonic streams. They found "all-residue substitution" to be more effective than "own-residue substitution."

For interfacing with ASR, they propose:

  • Re-training ASR Parameters: The Hidden Markov Model (HMM) parameters of the ASR system are re-trained using data that has been processed through the Bi-HBSS and binauralized using Head-Related Transfer Functions (HRTFs) to mimic the effect of the segregated sound arriving at the ears from specific directions.
  • Power Adjustment: The power of the segregated sound is adjusted to match the original sound's power.

Datasets

The paper describes experiments using:

  • Japanese Vowels and Utterances: Used for initial benchmarking of the segregation system.
  • ATR Speech Database: A standard database used for training and testing the ASR system. This includes male and female speaker data.
  • Custom Benchmarks:
    • Benchmarks 1-3: Mixtures of two women's utterances of a word.
  • Benchmarks 4-5: Mixtures of two sounds, including an intermittent sound, to test segregation robustness.
  • 500 Two-Word Testing Combinations: Used for the main experiment involving two speakers.

Key Insights and Ideas

  • Binaural Cues are Crucial for Segregation: Incorporating directional information (ITD/IID) significantly improves the ability to segregate sound streams, especially when harmonic structures are similar.
  • HRTFs Introduce Significant ASR Degradation: While binaural processing is good for segregation, the HRTF transformation itself can severely distort the signal, leading to a substantial drop in ASR accuracy. This was a key finding not previously highlighted.
  • Re-training ASR is Key to Recovery: Simply applying segregation isn't enough; the ASR system needs to be adapted. Re-training the HMM parameters with HRTF-processed data proved effective in recovering much of the lost recognition performance.
  • Consonants are Handled Implicitly: The "residue substitution" approach, while not explicitly extracting consonants, effectively reconstructs them by filling in the gaps left by harmonic stream extraction.
  • "Prince Shotoku Effect" is Achievable: The system demonstrates a computer's ability to "listen" to multiple sounds simultaneously by segregating them, a capability inspired by the legend of Prince Shotoku.

Key Results

  • Segregation Performance: The Bi-HBSS system, with D-grouping and all-residue substitution, showed improved segregation compared to HBSS alone.
  • ASR Degradation by HRTF: The cumulative accuracy for the 10th candidate dropped by 11.4% to 30.1% due to HRTF processing, depending on the sound source direction.
  • ASR Recovery: Re-training the HMM with HRTF-processed data (HMM-hrtf-m) significantly reduced this degradation, making it almost vanish for certain directions.
  • Main Experiment (Two Speakers): For 500 mixtures of two women's utterances, the cumulative accuracy of word recognition up to the 10th candidate was, on average, 75%. This is a significant improvement over performance without segregation, where recognition was near zero for the second speaker.

Limitations and Future Work

  • HRTF Distortion: The paper acknowledges that HRTF processing still introduces spectral distortion, and they are working on more sophisticated mechanisms to integrate HBSS and Bi-HBSS to overcome these drawbacks.
  • Consonant Extraction: The current method for handling consonants is an approximation based on auditory induction. Future work should incorporate more specific psychoacoustic phenomena like formants, AM/FM modulations, and more advanced localization cues.
  • Temporal Characteristics: The F-grouping method was found to be poor because it only used the previous state of fundamental frequency. Exploiting longer temporal characteristics is suggested for future work.
  • Top-Down Processing: The paper notes that while their approach is bottom-up, top-down processing (using listener knowledge) is also essential for CASA and should be explored.
  • HMM Limitations: The current assessment uses standard HMMs. Future work should evaluate the proposed segregation in more advanced HMM frameworks.
  • Moving Sound Sources: The F-Char correction method for HRTF distortion is not suitable for moving sound sources, making the re-trained HMM approach more practical.

Summary updated

BibTeX
@inproceedings{1996interfacing,
  title = {{Interfacing Sound Stream Segregation to Automatic Speech Recognition — Preliminary Results on Listening to Several Sounds Simultaneously}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1996},
}
Figure