Conformer-Based Target-Speaker Automatic Speech Recognition For Single-Channel Audio

Yang Zhang, Krishna C. Puvvada, Vitaly Lavrukhin, Boris Ginsburg

TL;DR

This paper introduces CONF-TSASR, a novel end-to-end system for transcribing a specific target speaker's speech from a mixed audio signal, even when other speakers are present. By combining a powerful Conformer architecture with a new spectrogram reconstruction loss, they achieve state-of-the-art performance on challenging datasets and set new benchmarks for target-speaker ASR.

What Problem Does This Paper Address?

Imagine you're in a busy cafe, and you want to transcribe only your conversation, ignoring everyone else. That's the core challenge of Target-Speaker Automatic Speech Recognition (TS-ASR). Existing methods either try to separate all speakers first (which can be suboptimal for transcription) or require knowing the profiles of all speakers in advance. This paper tackles the problem of TS-ASR using only a single-channel audio input and a profile of the target speaker (provided as a short, clean audio sample). This makes it ideal for scenarios where you only care about one person's speech in a noisy, multi-speaker environment.

Methods

The CONF-TSASR model is built upon the SpeakerBeam architecture and comprises three key modules:

  • TitaNet: This module extracts a compact speaker embedding from the provided auxiliary utterance (the target speaker's clean speech sample). This embedding acts as the "profile" for the target speaker.
  • Conformer-based MaskNet: This network takes the mixed audio's features and the speaker embedding to generate a time-frequency mask. This mask is designed to isolate the target speaker's speech components.
  • Conformer ASR Module: Finally, a Conformer-based Automatic Speech Recognition (ASR) module transcribes the masked speech features, effectively producing the target speaker's transcript.

The entire system is trained end-to-end using two loss functions:

  1. Connectionist Temporal Classification (CTC) loss: A standard loss for sequence-to-sequence tasks like ASR.
  2. Spectrogram Reconstruction Loss: A novel addition that encourages the model to better separate the target speaker's spectrogram from the mixture. This loss is scale-invariant (SiSNR), meaning it focuses on the relative structure of the spectrograms.

The MaskNet and ASR modules are based on the powerful Conformer architecture, known for its effectiveness in speech recognition.

Datasets

The authors evaluate their model on several challenging datasets:

  • WSJ0-2mix-extr: A standard dataset for two-speaker mixtures.
  • WSJ0-3mix-extr: A new benchmark for three-speaker mixtures.
  • LibriSpeech2Mix: A two-speaker mixture dataset derived from LibriSpeech.
  • LibriSpeech3Mix: A new benchmark for three-speaker mixtures derived from LibriSpeech.

They report state-of-the-art results on WSJ0-2mix-extr and establish new benchmarks for WSJ0-3mix-extr and the LibriSpeechMix datasets.

Key Insights and Ideas

  • The power of Conformer for TS-ASR: Integrating the Conformer architecture into both the masking and transcription modules significantly boosts performance, outperforming previous approaches.
  • Spectrogram reconstruction loss is crucial: Adding this novel loss function leads to substantial improvements, especially in more challenging three-speaker scenarios, highlighting its effectiveness in enhancing source separation for ASR.
  • End-to-end training is key: Jointly optimizing speaker embedding, masking, and transcription allows the model to learn a more cohesive and effective system compared to cascaded approaches.
  • Single auxiliary utterance is sufficient: The model demonstrates strong performance even with just one short, clean utterance of the target speaker, making it practical for real-world applications.

Key Results

The CONF-TSASR model achieves state-of-the-art Target-Speaker Word Error Rate (TS-WER) on WSJ0-2mix-extr, reaching 4.2% when trained with the spectrogram reconstruction loss. It also sets new benchmarks for:

  • WSJ0-3mix-extr: 12.4% TS-WER.
  • LibriSpeech2Mix: 4.2% TS-WER (for two-speaker mixtures).
  • LibriSpeech3Mix: 7.6% TS-WER (for three-speaker mixtures).

The paper also shows that the model's performance is sensitive to the Signal-to-Noise Ratio (SNR), particularly in three-speaker mixtures.

Limitations and Future Work

  • Computational Cost: While non-autoregressive, the model still requires one inference pass per speaker if one needs to transcribe multiple target speakers from the same audio.
  • Auxiliary Utterance Length: While one auxiliary utterance is sufficient, the paper notes a slight performance drop when using a shorter auxiliary utterance (7.5 sec vs. longer ones), suggesting that robust speaker profiles are important.
  • Future Work: The authors plan to open-source their model through the NVIDIA NeMo toolkit, which will facilitate further research and adoption. Future work could explore extending this to streaming scenarios or handling an even larger number of interfering speakers.

Summary updated

BibTeX
@inproceedings{zhang2023conformer,
  author = {Yang Zhang and Krishna C. Puvvada and Vitaly Lavrukhin and Boris Ginsburg},
  title = {{Conformer-Based Target-Speaker Automatic Speech Recognition For Single-Channel Audio}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095115},
}
Figure