Probabilistic Permutation Invariant Training for Speech Separation

Midia Yousefi, Soheil Khorram, John H.l. HansenInterspeech 2019

TL;DR

This paper introduces Probabilistic Permutation Invariant Training (Prob-PIT), a novel approach to speech separation that addresses the "label permutation ambiguity" problem. By treating output-label assignments as a probabilistic latent variable, Prob-PIT offers a more robust training method than traditional Permutation Invariant Training (PIT), leading to significantly improved separation performance.

What Problem Does This Paper Address?

Speech separation, the task of isolating individual speech signals from a mixed audio stream, is a long-standing challenge. A key hurdle in training neural networks for this task is "label permutation ambiguity." When a network is designed to separate S sources, it can produce S! different output permutations. For example, if the network outputs "Speaker A" and "Speaker B," it's unclear whether the first output corresponds to the actual Speaker A or Speaker B. Traditional Permutation Invariant Training (PIT) tackles this by selecting the output-label assignment that minimizes the separation error. However, this hard decision can be problematic, especially in the early stages of training when the network's outputs are unreliable. This can lead to overconfident, incorrect assignments that hinder effective learning. This paper aims to improve upon PIT by introducing a more flexible, probabilistic approach to handle this ambiguity.

Methods

The core innovation is Probabilistic Permutation Invariant Training (Prob-PIT). Instead of making a hard choice for the best output-label assignment, Prob-PIT treats the permutation as a discrete latent random variable with a uniform prior distribution. This means all possible permutations are considered equally likely a priori.

The training objective is to maximize the log-likelihood of the observed data given the model. This log-likelihood is defined over all possible permutations, taking into account their separation errors and prior probabilities. Mathematically, this is achieved by replacing the minimum function in PIT with a "soft-minimum" function. This soft-minimum allows the gradients from all permutations to contribute to the training process, with the contribution weighted by their respective costs. The paper shows that standard PIT is a special case of Prob-PIT when the smoothing factor (γ) of the soft-minimum is zero. By increasing γ, the model pays more attention to all permutations, leading to a smoother optimization landscape and potentially avoiding poor local minima.

The model architecture used is a standard LSTM-based network, common in speech separation tasks, consisting of a feed-forward layer followed by two recurrent LSTM layers.

Datasets

The proposed Prob-PIT method was evaluated on two datasets:

  • TIMIT: A widely used dataset for speech research.
  • GRID: A multi-speaker, sentence corpus often used for speech separation and recognition tasks.

For both datasets, the authors generated mixed speech by artificially summing random sentences from randomly chosen speakers. They created training, validation, and test sets with specific sizes for each.

Key Insights and Ideas

  • The "overconfident" nature of PIT's hard decision is a significant drawback: Especially early in training, when network outputs are noisy, forcing a single "best" assignment can lead to incorrect learning signals.
  • Probabilistic assignment smooths the optimization landscape: By considering all permutations with varying degrees of influence (controlled by γ), Prob-PIT avoids getting stuck in suboptimal solutions that a hard minimum might lead to.
  • Soft-minimum is key: Replacing the hard minimum with a soft-minimum function is the practical implementation that allows for probabilistic consideration of all permutations.
  • γ controls the trade-off: The parameter γ in the soft-minimum function allows for a tunable balance between relying on the single best permutation (low γ) and considering all permutations (high γ).

Key Results

Prob-PIT consistently outperformed the conventional PIT baseline across both the TIMIT and GRID datasets. The improvements were measured using Signal to Distortion Ratio (SDR) and Signal to Interference Ratio (SIR).

  • Significant performance gains: Prob-PIT showed statistically significant improvements (p-value < 0.01) over PIT for specific ranges of γ.
  • Optimal γ values found: The best separation performance was achieved with γ = 45 for the GRID dataset and γ = 32 for the TIMIT dataset.
  • Reduced variance: For reasonable choices of γ, the variance in SDR and SIR values was lower, indicating a more stable and reliable system.

Limitations and Future Work

  • Computational cost: While not explicitly detailed as a limitation, considering all permutations inherently increases computational complexity compared to PIT's single-best selection. The paper mentions that Prob-PIT can be implemented by replacing the minimum function with a soft-minimum, suggesting a trade-off between performance and computation.
  • Optimal γ selection: The paper explores different values of γ to find the optimal one for each dataset, implying that this parameter might need tuning for different scenarios.
  • Future work: The paper doesn't explicitly detail future work, but the success of Prob-PIT suggests further exploration into its application in more complex multi-speaker scenarios and its potential for real-time processing. The availability of the code on GitHub also facilitates further research by the community.

Summary updated

BibTeX
@inproceedings{yousefi2019probabilistic,
  author = {Midia Yousefi and Soheil Khorram and John H.L. Hansen},
  title = {{Probabilistic Permutation Invariant Training for Speech Separation}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure