Unconstrained speech separation by composition of longest segments

Ming Ji, Ramji Srinivasan, Danny CrookesInterspeech 2012

TL;DR

This paper introduces CLOSE, a novel data-driven approach to separating mixed speech signals without prior knowledge of the speakers' vocabulary or language. By focusing on matching the longest possible segments of mixed speech to clean training data, CLOSE significantly improves separation accuracy and subsequent speech recognition performance, especially in challenging low-signal-to-noise conditions.

What Problem Does This Paper Address?

Separating individual speech signals from a mixture (like a conversation with multiple people talking at once) is a fundamental challenge in audio processing. Traditionally, this has been approached in two ways:

  1. Constrained Speech Separation: This assumes we know a lot about the speech beforehand – its vocabulary, grammar, and even the speakers' identities. This prior knowledge acts like a strong guide, leading to near-human performance in some cases.
  2. Unconstrained Speech Separation: This is the harder problem. Here, we have no prior knowledge about the speakers, their language, or their speaking styles. Existing methods for unconstrained separation often treat speech frame-by-frame or capture only short-term dynamics, which can lead to significant errors, especially when speakers overlap for extended periods or when the signal is noisy.

This paper tackles the unconstrained speech separation problem, aiming to achieve the high performance of constrained methods without the restrictive prior knowledge. The core challenge is to impose useful temporal constraints on the separation process even when the speech dynamics are unknown and arbitrary.

Methods

The paper proposes a new approach called CLOSE (Composition of Longest Segments). The key idea is to move beyond frame-by-frame processing and instead identify and match segments of consecutive speech frames.

Here's how it works:

  • Modeling Training Utterances: For each speaker, they first train a Gaussian Mixture Model (GMM). Then, each training utterance is represented as a sequence of Gaussian components, effectively capturing the temporal dynamics of that specific utterance.
  • Matching Longest Segments: When presented with a mixed speech signal, CLOSE searches for the longest possible segments of this mixed speech that can be accurately matched by combining segments from the clean training data of the two speakers.
  • Why Longest Segments Matter: The intuition is that longer matched segments provide more specific and reliable information about the underlying speech. This reduces uncertainty and, consequently, the error in separating the individual speech signals.
  • Two-Sided vs. One-Sided: The paper describes a "two-sided" approach where both constituent training segments are constrained by their longest matches. They also introduce a "one-sided" simplification, where one constituent segment is constrained, and the other is less constrained. The experiments report results from the one-sided approach, which offers a good balance between performance and computational complexity.
  • Reconstruction: After identifying the longest matching segments, the individual speech frames for each speaker are reconstructed using a weighted average of the corresponding Gaussian components from the matched training segments. This smoothing across segments helps improve accuracy and robustness.

Datasets

  • Training Data: The Wall Street Journal (WSJO) database was used. Specifically, 20 speakers (10 male, 10 female) were selected, with about 140 utterances per speaker. The majority of these utterances were used for training the speaker models.
  • Test Data: For testing, two utterances from each speaker were mixed with utterances from other speakers at various target-to-masker ratios (TMRs) ranging from 10 dB to -10 dB. This created 760 mixed speech utterances for separation. The test utterances were chosen such that they had no sentences in common with the training data, simulating an unconstrained scenario.

Key Insights and Ideas

  • The Power of Long-Range Temporal Constraints: The paper strongly suggests that imposing long-range temporal constraints, even in an unconstrained setting, is crucial for improving speech separation. CLOSE achieves this by treating segments as units rather than individual frames.
  • "Longest Match" as a Novel Constraint: The core innovation is using the "longest matching segment" criterion. This is a data-driven way to find reliable correspondences between mixed and clean speech, reducing ambiguity.
  • Improved Robustness to Noise: The results show that CLOSE significantly outperforms traditional GMM-based methods, especially at lower TMRs (i.e., when the target speech is more obscured by background noise). This indicates greater robustness.
  • Direct Impact on Speech Recognition: The paper demonstrates that better speech separation directly translates to improved performance in a downstream task like large-vocabulary continuous speech recognition.

Key Results

  • Objective Measures: CLOSE significantly improved segmental Signal-to-Noise Ratio (SNR) and Perceptual Evaluation of Speech Quality (PESQ) scores compared to a standard GMM-based separation system, particularly at lower TMRs.
  • Subjective Measures: Mean Opinion Scores (MOS) for quality and intelligibility, as well as informal preference tests, showed that listeners consistently preferred speech separated by CLOSE over that separated by the GMM system. At -10 dB TMR, CLOSE achieved an average intelligibility score of 4.5 (between "mostly intelligible only if I concentrate" and "mostly intelligible"), while the GMM system scored 2.6 (between "slightly intelligible" and "somewhat intelligible").
  • Speech Recognition Accuracy: In large-vocabulary continuous speech recognition tests, CLOSE reduced word errors by over 50% compared to the GMM system, especially at lower TMRs. For example, at -10 dB TMR, CLOSE achieved a word accuracy of 22.1%, compared to 4.2% for the GMM system.
  • Computational Feasibility: The CLOSE algorithm was found to be computationally feasible, taking on average 2.6 times longer than the GMM-based algorithm for separating two clean utterances from a mixture.

Limitations and Future Work

  • Computational Cost: While feasible, CLOSE is still more computationally intensive than frame-by-frame methods. The paper mentions reducing complexity by considering a "one-sided" approach.
  • Speaker Pair Identification: The paper assumes that a speaker-pair identification algorithm can be used to identify the two speakers in the mixture. The performance of CLOSE would depend on the accuracy of this initial step.
  • Future Work: The paper doesn't explicitly detail future work, but the success in unconstrained separation suggests potential applications in more complex multi-speaker scenarios and further exploration of optimizing the segment matching and reconstruction processes. The authors also point to the computational feasibility as a positive aspect for practical deployment.

Summary updated

BibTeX
@inproceedings{ji2012unconstrained,
  author = {Ming Ji and Ramji Srinivasan and Danny Crookes},
  title = {{Unconstrained speech separation by composition of longest segments}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure