Enhancing and Adversarial: Improve ASR with Speaker Labels

Wei Zhou, Wei Zhou 0043, Haotian Wu, Jingjing Xu, Mohammad Zeineldeen, Christoph Lüscher, Ralf Schlüter, Hermann NeyICASSP 2023 · DOI

TL;DR

This paper introduces novel multi-task learning techniques to improve Automatic Speech Recognition (ASR) by leveraging speaker information. They propose an adaptive gradient reversal layer for more stable adversarial training and demonstrate that strategically applying speaker-enhancing and adversarial training at different layers of a conformer model leads to significant improvements in ASR performance.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle with speaker variability, meaning their performance can degrade when faced with different speakers. While techniques like multi-task learning (MTL) have been used to address this, they often involve conflicting objectives: "domain enhancing" aims to make the model more sensitive to speaker differences (potentially for speaker identification), while "domain adversarial" aims to make the model less sensitive to speaker differences (for better generalization). The authors explore how to best apply these opposing objectives, specifically using speaker labels, to improve ASR performance. They also aim to achieve these improvements without relying on complex speaker embeddings like i-vectors, which can add runtime overhead.

Methods

The core of this work lies in applying two MTL objectives with speaker labels to a conformer-based ASR model:

  • Speaker Enhancing Training (DET): This objective aims to boost the model's ability to distinguish speakers. It's achieved by adding a speaker classifier to an intermediate layer of the ASR network and training it to maximize the probability of the correct speaker given the layer's output.
  • Speaker Adversarial Training (DAT): This objective aims to make the ASR model speaker-invariant. It uses a Gradient Reversal Layer (GRL) to train a speaker classifier to minimize the probability of identifying the speaker from an intermediate layer's output. This forces the ASR network to learn representations that are less dependent on speaker characteristics.

A key novelty is the Proposed Adaptive GRL. Standard GRLs can be difficult to tune and may lead to unstable training. The authors introduce an adaptive GRL that dynamically scales the adversarial gradient based on the speaker classifier's confidence. This allows for more aggressive adversarial training when the classifier is confident (indicating speaker variance) and less when it's not, leading to more stable and effective training with less manual tuning.

They also investigate combining these techniques by applying enhancing training to lower layers of the conformer network (where speaker information is thought to be more prominent) and adversarial training to higher layers (where content information is more dominant).

Datasets

  • Switchboard (SWB): A 300-hour corpus of conversational telephone speech, used for most experiments and evaluation.
  • Librispeech (LBS): A larger corpus (960 hours) of read English speech, used for generalization testing.
  • Hub5'00 and Hub5'01: Development and test sets derived from the Switchboard corpus.
  • CallHome (CH): A noisier subset of Hub5'00, used to evaluate robustness.

Key Insights and Ideas

  • Strategic Layer Placement Matters: The authors found that applying speaker enhancing training to lower layers and speaker adversarial training to higher layers of the conformer network yields the best results. This aligns with the intuition that lower layers capture more speaker-specific cues, while higher layers should focus on content-invariant representations for ASR.
  • Adaptive GRL for Stability: The proposed adaptive GRL significantly simplifies adversarial training by removing the need for extensive hyperparameter tuning, making the technique more practical.
  • Synergy of Enhancing and Adversarial: While simply combining both objectives in one go didn't yield further improvements, sequentially applying them (enhancing first, then adversarial fine-tuning) proved effective, suggesting a delicate balance is needed.
  • Speaker Embeddings vs. Learned Representations: While i-vectors (a form of speaker embedding) provide strong improvements, the speaker-based MTL achieves comparable performance without the added complexity of explicit speaker embeddings, which is a significant practical advantage.
  • NN Architecture Influence: The effectiveness of speaker-based MTL is shown to be dependent on the ASR model's capacity. Weaker NNs (like BLSTM) showed less benefit compared to stronger conformer models.

Key Results

  • The best speaker-based MTL approach achieved a 7% relative improvement in Word Error Rate (WER) on the Switchboard Hub5'00 dataset compared to the baseline.
  • This performance was comparable to using i-vectors with adversarial training.
  • The proposed adaptive GRL demonstrated stable training behavior, as evidenced by the loss curves.
  • The proposed methods showed generalization to the Librispeech dataset, with the noisier CallHome subset benefiting the most.

Limitations and Future Work

  • Weaker NNs: The paper notes that the benefits of speaker-based MTL are less pronounced on weaker ASR architectures like BLSTM, suggesting that the model needs sufficient capacity to learn speaker-aware representations.
  • Clean Data: Improvements were smaller on cleaner datasets like Librispeech compared to noisier datasets like Switchboard, indicating that the techniques are particularly effective at handling real-world variability.
  • Sequential Application: While sequential application of enhancing and adversarial training was beneficial, further research could explore more sophisticated ways to combine these objectives.
  • Hyperparameter Tuning: Although the adaptive GRL reduces tuning effort, some parameters (like the weighting of the MTL losses) still require careful consideration.

Summary updated

BibTeX
@inproceedings{zhou2023enhancing,
  author = {Wei Zhou and Wei Zhou and Haotian Wu and Jingjing Xu and Mohammad Zeineldeen and Christoph Lüscher and Ralf Schlüter and Hermann Ney},
  title = {{Enhancing and Adversarial: Improve ASR with Speaker Labels}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096722},
}
Figure