PGSS: Pitch-Guided Speech Separation

Xiang Li, Yiwen Wang, Yifan Sun, Xihong Wu, Jing ChenAAAI 2023 · DOI

TL;DR

This paper introduces PGSS, a novel pitch-guided speech separation framework that leverages the psychoacoustic principle of pitch priming to address the challenges of permutation and unknown speaker numbers in monaural speech separation. By using pitch contours as a conditioning signal, PGSS achieves improved separation performance and perceptual quality, offering a more human-like approach to disentangling overlapping speech.

What Problem Does This Paper Address?

The "cocktail party problem" – the human ability to focus on one voice amidst a cacophony of sounds – is a long-standing challenge in artificial intelligence. Monaural speech separation aims to replicate this by separating individual speech signals from a single audio recording. Existing deep learning models often struggle with two key issues: the "permutation problem," where the model can't consistently assign an output to a specific speaker across different training examples, and the "unknown speaker number problem," where the model architecture is fixed for a predetermined number of speakers. While techniques like permutation invariant training (PIT) and deep clustering (DC) have been developed, they don't fully resolve the unknown speaker number issue. This paper proposes a solution inspired by auditory scene analysis (ASA), specifically the concept of "pitch priming," where a listener's attention can be guided by the pitch of a target speaker.

Methods

The proposed PGSS framework operates in two distinct phases:

  • Phase I: Multi-Speaker Pitch Extraction: This phase models the bottom-up perceptual cue extraction in ASA. It employs a novel two-stage approach:
  • Frame-Level Pitch Estimation: This stage treats pitch extraction as a multi-label classification task, predicting pitch states for each time-frequency unit of the mixture spectrogram. It uses a 2D CNN for harmonic structure analysis followed by a linear layer. This approach is speaker-number independent.
  • Utterance-Level Pitch Tracking: This stage takes the frame-level pitch candidates and links them to specific speakers, forming continuous pitch contours. It uses a conditional chain (cond-chain) model, essentially a seq2Mseq mapping problem, to track these contours across time. A stop criterion is used to handle variable numbers of output sequences.
  • Phase II: Pitch-Conditional Speech Separation: This phase models the effect of pitch priming. A selected pitch contour from Phase I is used as a conditioning signal for a Conditional Generative Adversarial Network (CGAN), specifically a Pitch-CGAN. This CGAN is inspired by the MelGAN architecture, which is known for generating high-quality waveforms. The Pitch-CGAN consists of:
  • Generator (G): Composed of G1 (an encoder-decoder network that produces target magnitude conditioned on pitch and mixture spectrogram) and G2 (a MelGAN-like structure that synthesizes the target waveform from the magnitude).
  • Discriminator: Multiple discriminators at different frequency scales are used to ensure the generated speech is realistic.

The framework avoids traditional time-frequency masking by directly producing the waveform, aiming for improved perceptual quality.

Datasets

The experiments were conducted on the Wall Street Journal (WSJ0) corpus. Specifically, the WSJ0-2mix and WSJ0-3mix datasets were used. These datasets are benchmarks for speech separation, created by mixing single-speaker utterances from the WSJ0 training and validation sets at various Signal-to-Noise Ratios (SNRs) ranging from 0 dB to 5 dB.

Key Insights and Ideas

  • Pitch as a powerful cue: The paper strongly advocates for pitch as a primary, low-level feature for guiding speech separation, drawing parallels to human auditory perception and the concept of pitch priming.
  • Decoupling pitch extraction and separation: The two-phase approach allows for specialized optimization of each component, potentially leading to better overall performance.
  • Avoiding permutation and unknown speaker number issues: By conditioning separation on a specific pitch contour, the model inherently handles these problems, as it's tasked with separating one speaker at a time.
  • Direct waveform generation for quality: Using a MelGAN-inspired generator for waveform synthesis, rather than relying on phase estimation or traditional masking, leads to improved perceptual quality.
  • Pitch's advantage for similar timbres: Pitch can remain discriminative even when speakers have similar timbres, a scenario where speaker embeddings might struggle.

Key Results

  • Pitch Extraction: The frame-level pitch estimation achieved high precision (93.4%) and recall (89.2%). Utterance-level pitch tracking showed competitive performance, with the proposed cond-chain method outperforming a baseline (uPIT-Pitch) across various SNRs and gender combinations.
  • Speech Separation:
    • The Pitch-CGAN with the MelGAN module (G2) significantly improved perceptual metrics (STOI and PESQ) compared to a version without it.
  • The proposed Pitch-CGAN system outperformed existing speaker-independent systems like uPIT and DPCL in terms of SDRi, PESQ, and STOI.
  • It also achieved superior results compared to speaker-dependent systems that rely on speaker embeddings.
  • On the WSJ0-3mix dataset, Pitch-CGAN achieved the highest scores across all metrics, demonstrating its effectiveness with more than two speakers.

Limitations and Future Work

  • Dependency on pitch estimation accuracy: The paper acknowledges that the accuracy of pitch estimation can impact the final separation performance. Errors in pitch tracking might be more detrimental than errors in speaker embeddings.
  • Multi-stage approach: While beneficial for explainability, the multi-stage nature of the framework means it's not end-to-end, which could be a limitation.
  • Future Work: The authors suggest exploring ways to combine the pitch extraction and separation phases into a more iterative, jointly optimized system. They also propose evaluating the framework with SOTA model architectures to assess its flexibility and potential for further improvement.

Summary updated

BibTeX
@inproceedings{li2023pgss,
  author = {Xiang Li and Yiwen Wang and Yifan Sun and Xihong Wu and Jing Chen},
  title = {{PGSS: Pitch-Guided Speech Separation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26542},
}
Figure