Self-Supervised Learning-Based Source Separation for Meeting Data

Yuang Li, Xianrui Zheng, Philip C. Woodland

TL;DR

This paper bridges the gap between self-supervised learning (SSL) for speech separation and its real-world application in meeting transcription. By rigorously evaluating seven SSL models on both simulated and real meeting data, they identify WavLM as the top performer and integrate it into an ASR system using a novel iterative source selection method, achieving significant improvements in transcription accuracy.

What Problem Does This Paper Address?

In multi-party conversations, speech often overlaps, making it difficult for Automatic Speech Recognition (ASR) systems to accurately transcribe what's being said. Source separation aims to disentangle these overlapping speech signals into individual speaker streams, which can then be fed into an ASR model for better performance. While self-supervised learning (SSL) has shown great promise for speech tasks, most research has focused on simulated datasets. This paper addresses the critical need to evaluate and adapt these SSL models for the challenging domain of real-world meeting data and to integrate them effectively into a complete transcription pipeline.

Methods

The core of this work involves two main components:

  1. SSL Model Evaluation and Integration: The authors compare seven prominent SSL models (TERA, Wav2Vec2, HuBert, UniSpeech-SAT, UniSpeech-SAT+, WavLM, WavLM+) for source separation. They use a two-phase fine-tuning strategy where a lightweight Conformer block is first trained on SSL representations, followed by fine-tuning the SSL model itself. For real-world adaptation, they employ Mixture Invariant Training (MixIT), specifically a time-frequency domain adaptation using phase-sensitive masking (PSM), to train with real overlapped speech.
  2. Iterative Source Selection: To select the desired speaker's stream from the multiple outputs of the separation model, a novel iterative source selection method is proposed. This method leverages speaker embeddings derived from diarization outputs. It iteratively refines average speaker embeddings by removing outliers and then selects the output source with the highest cosine similarity to the refined average embedding. This approach avoids the need for enrollment audio and is designed to be plug-and-play within a transcription system.

Datasets

  • Simulated Datasets:
    • R-LibriMix: An extension of LibriMix, augmented with simulated reverberations and isotropic noise.
  • LibriCSS: A dataset for continuous speech separation.
  • Real-World Datasets:
    • AMI: The Audio-Visual Multi-modal Meeting Corpus, used for both ASR model fine-tuning and evaluation of the complete transcription system.
  • Benchmark: The paper implicitly contributes to a benchmark for evaluating SSL models on real meeting data, highlighting the performance discrepancies compared to simulated data.

Key Insights and Ideas

  • Real-world data is crucial: Performance on simulated datasets doesn't always translate to real-world meeting data. Models that generalize better to real-world conditions are preferred.
  • Utterance mixing is key: SSL models that incorporate utterance mixing augmentation (like UniSpeech-SAT and WavLM) show significantly better performance for source separation, suggesting exposure to multiple speakers during pre-training is highly beneficial.
  • Fine-tuning SSL models matters: Unfreezing and fine-tuning the SSL models themselves, not just a downstream head, leads to substantial performance gains, allowing them to learn task-specific representations without overfitting.
  • MixIT for real data: Adapting unsupervised MixIT to the time-frequency domain with PSM is effective for training with real overlapped speech, overcoming limitations of purely synthetic training.
  • Iterative source selection is effective: The proposed iterative source selection method provides a practical way to extract a target speaker's signal without modifying the core separation architecture, and it works well even with noisy diarization outputs.

Key Results

  • SSL Model Performance: WavLM (specifically WavLM+) emerged as the best-performing SSL model for source separation across R-LibriMix, LibriCSS, and AMI datasets, achieving the lowest word error rates (WERs) after fine-tuning.
  • Transcription System Improvement: When the WavLM-based source separation and iterative source selection were integrated into an ASR system (using a fine-tuned W2V2-AMI-Sep model), absolute reductions of 1.9% and 1.5% in concatenated minimum-permutation word error rate (cpWER-us) were observed on the AMI dev and test sets, respectively.
  • MixIT Benefits: Semi-supervised training with MixIT improved WERs by about 1% on AMI compared to pure PIT, demonstrating the value of using real mixtures.
  • Source Selection Accuracy: The iterative source selection achieved an accuracy of around 85% on AMI, which is slightly lower than pure PIT with two sources (over 90%), highlighting a trade-off between separation complexity and selection accuracy.

Limitations and Future Work

  • Limitations:
    • The paper notes that while MixIT is effective, it can lead to over-separation, and the selection accuracy can be lower with more output sources.
  • The performance of the transcription system is sensitive to the quality of the ASR model and its fine-tuning strategy.
  • Future Work:
    • The authors suggest that the iterative source selection method's robustness to noisy diarization outputs could be further investigated.
  • Exploring more advanced unsupervised or semi-supervised training techniques for source separation on real-world data remains an open area.
  • Further research into optimizing the interaction between source separation and ASR models, especially when the ASR model has already been exposed to overlapped data, is warranted.

Summary updated

Abstract

Source separation can improve automatic speech recognition (ASR) under multi-party meeting scenarios by extracting single-speaker signals from overlapped speech. Despite the success of self-supervised learning models in single-channel source separation, most studies have focused on simulated setups. In this paper, seven SSL models were compared on both simulated and real-world corpora. Then, we propose to integrate the best-performing model WavLM into an automatic transcription system through a novel iterative source selection method. To improve real-world performance, time-domain unsupervised mixture invariant training was adapted to the time-frequency domain. Experiments showed that in the transcription system when source separation was inserted before an ASR model fine-tuned on separated speech, absolute reductions of 1.9% and 1.5% in concatenated minimum-permutation word error rate for an unknown number of speakers (cpWER-us) were observed on the AMI dev and test sets.
BibTeX
@inproceedings{li2023self,
  author = {Yuang Li and Xianrui Zheng and Philip C. Woodland},
  title = {{Self-Supervised Learning-Based Source Separation for Meeting Data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.48550/arxiv.2304.00871},
}
Figure