Real-Time End-to-End Monaural Multi-Speaker Speech Recognition

Li Song, Beibei Ouyang, Fuchuan Tong, Dexin Liao, Lin Li, Qingyang Hong2021 · DOI

TL;DR

This paper tackles the challenge of real-time multi-speaker speech recognition from a single audio channel. The authors propose an improved non-autoregressive decoding approach, Mask-CTC, that significantly speeds up recognition without sacrificing accuracy, making it practical for real-world applications.

What Problem Does This Paper Address?

Modern speech recognition systems have become very good at understanding single speakers in clean environments. However, real-world scenarios, like meetings or phone calls, often involve multiple people speaking simultaneously, sometimes overlapping. This "cocktail party problem" is notoriously difficult for speech recognition. While techniques exist to separate these overlapping voices (source separation) and then recognize them, most current end-to-end multi-speaker ASR systems rely on autoregressive decoding. This means they predict words one by one, which is inherently slow and hinders real-time applications. The paper aims to bridge this gap by developing an end-to-end system that can recognize multiple speakers from a single audio stream in real-time.

Methods

The paper builds upon existing end-to-end multi-speaker ASR architectures, categorizing them into two main types:

  1. Full E2E Multi-speaker ASR: This approach uses a single encoder for the mixed speech, followed by speaker-dependent encoders to implicitly separate speakers, and then shared recognition encoders/decoders.
  2. Combined Source Separation + ASR: This involves using a separate source separation model (like TF-masking or TasNet) to isolate individual speaker signals, which are then fed into a standard ASR system.

The core novelty of this paper lies in improving and applying a non-autoregressive decoding mechanism called Mask-CTC. Traditional attention-based decoders are autoregressive. Mask-CTC, inspired by masked language models (like BERT), aims for parallel decoding. The authors refine the original Mask-CTC by:

  • CTC Prefix Beam Search: Instead of greedy search for initial predictions, they use beam search to generate multiple candidate sequences, improving initial accuracy.
  • CMLM Decoder: A conditional masked language model (CMLM) is used for iterative refinement of these candidates.
  • Transformer Language Model Re-scoring: A separate Transformer language model is used to re-score the refined candidates, selecting the best final recognition output.

This modified Mask-CTC is then integrated into both the "Full E2E" and "Combined Source Separation + ASR" architectures, replacing the original autoregressive decoders.

Datasets

The primary dataset used for experiments is LibriMix. Specifically, the authors use the Libri2Mix subset, which is derived from LibriSpeech and includes mixtures of two speakers under various conditions (clean/noisy, different overlap ratios). They note that LibriMix is preferred over datasets like wsj0-2mix because its overlap ratios are more representative of real-world scenarios.

Key Insights and Ideas

  • Real-time is Crucial: The paper strongly emphasizes that autoregressive decoding, while effective, is a bottleneck for practical multi-speaker ASR. Achieving real-time performance is a key differentiator.
  • Non-autoregressive Decoding for Multi-speaker ASR: Adapting non-autoregressive models like Mask-CTC to the multi-speaker domain is a promising direction for speed.
  • Improved Mask-CTC Decoding Strategy: The combination of CTC prefix beam search, CMLM iterative refinement, and Transformer LM re-scoring significantly enhances the accuracy of the non-autoregressive approach.
  • Time-domain Separation Advantage: Experiments suggest that time-domain source separation methods (like TasNet) generally yield better results than frequency-domain methods, likely due to their ability to leverage both amplitude and phase information.
  • Joint Training Benefits: For combined separation and recognition systems, joint training with a weighted loss function (balancing separation and recognition objectives) is superior to cascade or purely ASR-driven approaches, which suffer from mismatch or training difficulties, respectively.

Key Results

The paper demonstrates that their modified Mask-CTC approach achieves real-time decoding (RTF < 1) in both the full E2E and combined separation+ASR architectures. Crucially, this speedup is achieved with recognition performance close to that of the autoregressive models. For instance, in Table 1, DPRNN-TasNet + modified Mask-CTC achieves a Min_test_WER of 15.19% with an RTF of 0.897, while autoregressive versions of similar models have much higher RTFs. The best overall performance was achieved by the DPRNN-TasNet + modified Mask-CTC combination.

Limitations and Future Work

  • Parameter Count: The authors mention that they aimed to keep the number of parameters close across models for fair comparison, but the exact trade-offs between parameter count and performance for their specific modified Mask-CTC are not deeply explored.
  • Dataset Size: While Libri2Mix is a good benchmark, the training set was reduced to 137 hours for efficiency. Larger datasets or more diverse acoustic conditions could further improve robustness.
  • Future Work: The paper implicitly suggests further exploration of non-autoregressive models and their integration into more complex multi-speaker scenarios. The success of the improved decoding strategy opens doors for further optimization of non-autoregressive ASR.

Summary updated

BibTeX
@misc{song2021real,
  author = {Li Song and Beibei Ouyang and Fuchuan Tong and Dexin Liao and Lin Li and Qingyang Hong},
  title = {{Real-Time End-to-End Monaural Multi-Speaker Speech Recognition}},
  year = {2021},
  doi = {10.21437/interspeech.2021-1449},
}
Figure