One Shot Learning for Speech Separation

Yuan-Kuei Wu, Kuan-Po Huang, Yu Tsao, Hung-yi Lee

TL;DR

This paper introduces meta-learning to speech separation, enabling models to quickly adapt to new speakers and acoustic environments with just one example. By leveraging MAML and ANIL algorithms with the Conv-TasNet architecture, they demonstrate significant improvements in generalization compared to traditional multi-task learning approaches.

What Problem Does This Paper Address?

Speech separation (SS) models have made great strides, but they often struggle when encountering new speakers or different acoustic conditions than those they were trained on. This lack of generalizability is a major hurdle, especially since SS is a crucial preprocessing step for many speech-related applications like recognition and identification. Building a single model that can handle every possible combination of speakers and environments is practically impossible due to the sheer scale and cost of data collection. The authors propose a more efficient solution: a model that can learn to adapt quickly to new scenarios with minimal data.

Methods

The core of this work lies in applying meta-learning techniques to speech separation. Specifically, they utilize two prominent meta-learning algorithms:

  • Model-Agnostic Meta-Learning (MAML): MAML aims to find a model initialization that is sensitive to changes and can be quickly fine-tuned for new tasks. It involves an "inner loop" for task-specific learning and an "outer loop" to update the meta-parameters.
  • Almost No Inner Loop (ANIL): ANIL is presented as a more computationally efficient variant of MAML. It focuses on "feature reuse," where the meta-initialization already contains useful information, and only task-specific layers are updated during adaptation.

These meta-learning algorithms are integrated with Conv-TasNet, a state-of-the-art time-domain audio separation network. Conv-TasNet comprises an encoder, a separator (which generates masks), and a decoder. The paper investigates which parts of Conv-TasNet act as feature-reuse layers and which are task-specific modules during the meta-learning process.

Datasets

The authors introduce new meta-learning datasets for speech separation:

  • WSJ0-2mix-meta: Used for training the meta-learning models. It's derived from the Wall Street Journal (WSJ0) corpus, creating tasks by mixing utterances from different speakers.
  • Libri-2mix-meta: Used for testing, derived from the LibriSpeech dataset.
  • VCTK-2mix-meta: Also used for testing, derived from the VCTK corpus.

These datasets are structured into "tasks," where each task involves mixing speech from a specific pair of speakers. The "support set" contains a single mixture for one-shot adaptation, and the "query set" is used for evaluating the adapted model. The datasets also incorporate noise from the MUSAN corpus for evaluating performance in noisy conditions.

Key Insights and Ideas

  • Meta-learning is effective for speech separation: The paper demonstrates that meta-learning can significantly improve a model's ability to adapt to unseen speakers and noisy environments, outperforming traditional multi-task learning.
  • One-shot adaptation is feasible: The proposed methods show that a model can learn to separate speech from new speakers with just a single example mixture.
  • The separator is the task-specific module: Experiments reveal that the separator component of Conv-TasNet is the most crucial part for task-specific adaptation, while the encoder and decoder act as feature-reuse layers. This insight is valuable for designing more efficient meta-learning architectures.
  • Rapid learning (MAML) might be superior to feature reuse (ANIL) for SS: While ANIL offers computational benefits, MAML (specifically, updating the whole model) showed better performance in their experiments, suggesting that rapid adaptation of all parameters is more beneficial for speech separation.

Key Results

  • Meta-learning models (MAML) consistently outperformed multi-task learning baselines on unseen speakers and noisy conditions.
  • For MAML, using a "best" pretrained model (trained for more epochs) generally yielded better results than a "half" pretrained model, though the performance drop was less severe than in multi-task learning, suggesting MAML is less sensitive to pretraining duration.
  • Comparing different fine-tuning strategies within multi-task learning, fine-tuning only the separator module (ANIL_s) showed better performance than fine-tuning the entire model or just the autoencoder.
  • However, when comparing meta-learning methods, MAML (specifically, updating the whole model) achieved the best SI-SNRi scores on both Libri-2mix-meta and VCTK-2mix-meta, with and without noise, indicating that rapid learning is the most effective approach for this task.

Limitations and Future Work

  • Model Agnosticism: The authors used Conv-TasNet as their SS model due to its strong performance and model-agnostic nature, which is compatible with MAML. However, they suggest that future work should explore applying these meta-learning methods to other SS models.
  • Further Exploration of ANIL: While ANIL was explored with different task-specific modules (separator vs. autoencoder), further investigation into its effectiveness and potential optimizations for speech separation could be beneficial.
  • Broader Applications: The paper focuses on two-speaker separation. Extending this to scenarios with more speakers or more complex acoustic environments could be a valuable next step.

Summary updated

Abstract

Despite the recent success of speech separation models, they fail to separate sources properly while facing different sets of people or noisy environments. To tackle this problem, we proposed to apply meta-learning to the speech separation task. We aimed to find a meta-initialization model, which can quickly adapt to new speakers by seeing only one mixture generated by those people. In this paper, we use model-agnostic meta-learning(MAML) algorithm and almost no inner loop(ANIL) algorithm in Conv-TasNet to achieve this goal. The experiment results show that our model can adapt not only to a new set of speakers but also noisy environments. Furthermore, we found out that the encoder and decoder serve as the feature-reuse layers, while the separator is the task-specific module.
BibTeX
@inproceedings{wu2021one,
  author = {Yuan-Kuei Wu and Kuan-Po Huang and Yu Tsao and Hung-yi Lee},
  title = {{One Shot Learning for Speech Separation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413956},
}
Figure