TL;DR
This paper introduces a novel speaker-independent approach to Single Channel Source Separation (SCSS) using sinusoidal features. By adapting existing speaker models to new speakers within a mixture, the method aims to improve separation performance without requiring pre-trained models for every individual. This is particularly useful for real-world scenarios where speaker identities are unknown.
What Problem Does This Paper Address?
Separating individual speech sources from a single audio channel (Single Channel Source Separation or SCSS) is a notoriously difficult problem, especially when the number of sources exceeds the number of available audio channels (the underdetermined case). Most successful methods rely on having pre-trained, speaker-specific models. However, in many practical applications, such as "computer audition" systems, the identities of the speakers in a mixture are unknown beforehand, making it impossible to use these pre-trained models. This paper tackles the challenge of achieving effective SCSS without prior knowledge of the speakers involved, aiming for a truly speaker-independent solution.
Methods
The core of the proposed approach builds upon the concept of Eigenvoice Adaptation (EA), extending it to utilize sinusoidal features. Here's a breakdown:
- Sinusoidal Features: Speech is represented as a sum of sinusoidal components, each characterized by amplitude, frequency, and phase. These features are extracted using the Short-Term Fourier Transform (STFT) and are adapted to the Mel scale to better align with human auditory perception.
- Speaker Codebooks: For each speaker, a "codebook" of these sinusoidal features is created. These codebooks act as a representation of a speaker's vocal characteristics.
- Eigenvoice Adaptation (EA): This technique, originally for speaker recognition, is adapted here. It involves creating a set of "eigenvectors" from the combined codebooks of multiple speakers. Any speaker's codebook can then be represented as a linear combination of these eigenvectors. For speaker-independent SCSS, the algorithm uses these eigenvectors to construct initial models for unknown speakers present in a mixture.
- Iterative Codebook Update: To refine the initial speaker models, an iterative algorithm is employed. This process repeatedly reconstructs the source using the current model and updates the model based on the extracted sinusoidal features of the reconstructed source. This helps to improve the accuracy of the separated signal.
- Successive Mixtures Codebook Replacement (SMCR): This novel enhancement leverages the fact that a single recording might contain multiple mixtures of the same speakers. Once a codebook for a dominant speaker is developed from one mixture, it's used as the starting point for separating subsequent mixtures involving the same speaker. This significantly speeds up the process and can lead to even better separation performance.
Datasets
- GRID Corpus: This corpus was used for both training and evaluation.
- Training Data: Codebooks were created for 20 speakers from the GRID corpus.
- Evaluation Data: 120 binary mixtures were created using utterances from four talkers not included in the training set. These mixtures were generated at seven different Signal-to-Signal Ratios (SSRs) ranging from -9 dB to +9 dB.
Key Insights and Ideas
- Bridging the Gap: The paper successfully bridges the gap between model-based SCSS and the need for speaker independence. By adapting existing speaker models to novel speakers, it offers a practical solution for real-world scenarios.
- Sinusoidal Features' Efficacy: The use of sinusoidal features, particularly when adapted to the Mel scale, proves effective for representing speaker characteristics in a way that can be leveraged for separation.
- Iterative Refinement Pays Off: The iterative codebook update algorithm demonstrates a clear improvement in separation quality, highlighting the benefit of refining initial estimates.
- Leveraging Sequential Data: The SMCR algorithm is a clever way to exploit the sequential nature of real-world audio, where speakers might appear in multiple mixtures, leading to computational efficiency and improved results.
Key Results
- Improved Separation: Experimental results, measured by Perceptual Evaluation of Speech Quality (PESQ) scores, indicate improved separation performance for the proposed speaker-independent approach compared to baseline methods.
- Dominant Speaker Performance: The SMCR strategy showed improvements in PESQ scores for the dominant speaker over the EA with iteration approach.
- Computational Efficiency: The SMCR approach is noted to be approximately 5 times faster than using the EA algorithm for each new mixed utterance.
- Limitations at Low SSRs: For negative SSRs (where the target speaker is less dominant), the proposed methods performed similarly, indicating a challenge in separating the less dominant source without prior knowledge.
Limitations and Future Work
- Less Dominant Speaker Reconstruction: A significant limitation acknowledged is the difficulty in accurately reconstructing the less dominant speaker's utterance. While the dominant speaker can be separated without prior SSR knowledge, reconstructing the less dominant speaker requires this information.
- Quantization Performance: The authors suggest that improving the quantization performance of the sinusoidal coders could further enhance separation quality.
- Future Directions:
- Addressing the reconstruction of the less dominant speaker.
- Exploring iterative approaches for further improvements.
- Investigating applications beyond speech, such as music-voice separation, using generic sinusoidal feature models.
- Adapting the method to more realistic scenarios, including reverberant environments and multiple simultaneous sound sources.
Summary updated