Learning Speaker Aware Offsets for Speaker Adaptation of Neural Networks

Leda Sari, Samuel Thomas, Mark A. Hasegawa-JohnsonInterspeech 2019 · OpenReview

Abstract

In this work, we present an unsupervised long short-term memory (LSTM) layer normalization technique called Adaptation by Speaker Aware Offsets (ASAO). These offsets are learned using an auxiliary network attached to the main senone classifier. The auxiliary network takes main network LSTM activations as input and reconstructs speaker, (speaker, phone), and (speaker, senone)-level averages of the activations by minimizing the mean-squared error. Once jointly trained, the network generates the offset itself during test time without requiring additional speaker information. Unlike many speaker adaptation studies that only target fully connected layers, ASAO is applicable to LSTM layers. Experiments on the HUB4 dataset show that ASAO achieves higher senone classification accuracy and lower word error rate (WER) than unadapted and adversarial models, with an absolute WER reduction of up to 2%.

Introduction: The Challenge of Speaker Variability

In automatic speech recognition (ASR), speaker variability—differences in accent, pitch, and vocal tract shape—creates a significant "mismatch" problem. A model trained on a general population often falters when encountering a new speaker. Traditional solutions typically involve feature-level normalization (like fMLLR) or supervised model-level adaptation. However, there is a growing need for unsupervised methods that can adapt complex sequence models like LSTMs on the fly.

Methodology: The ASAO Framework

The core innovation of this research is the ASAO framework, which introduces an auxiliary network to neutralize speaker-specific variability within the hidden layers of the main classifier.

The ASAO architecture showing the main network and the auxiliary network generating speaker-aware offsets.

How ASAO Works

  1. The Main Network: A standard acoustic state (senone) classifier utilizing LSTM and fully connected (FC) layers.
  2. The Auxiliary Network: This component "piggybacks" on the main network, taking hidden activations ($h$) from an LSTM layer as input.
  3. Targeting the Shift: During training, the auxiliary network is tasked with predicting the deviation of a speaker's average activation from the global average across speaker, phone, and senone levels.
  4. Normalization: The auxiliary network produces a bottleneck feature ($z$), which is transformed by a matrix ($T$) into a speaker-aware offset. This offset is subtracted from the main network’s activations: $\hat{h} = h - T(z)$. This process effectively "strips away" speaker-specific components, leaving a speaker-invariant signal for subsequent layers.

Experimental Setup

The researchers evaluated ASAO using the HUB4 Broadcast News dataset. They tested two primary conditions: a 150-speaker set (more data per speaker) and a 600-speaker set (less data per speaker). Evaluation was performed on a held-out set of speakers to ensure the adaptation was truly unsupervised and capable of generalizing to entirely new voices.

Key Insights and Visualization

Early Adaptation and Complementarity

The study found that adapting the first LSTM layer yielded the most significant gains. As signals move deeper into a network, they naturally become more focused on phonetic content rather than speaker identity. Furthermore, ASAO provided improvements even when input features were already normalized with fMLLR, suggesting it captures internal non-linear variations that linear methods miss.

Visualizing Speaker Invariance

To confirm the effectiveness of the normalization, the authors used Linear Discriminant Analysis (LDA) to project the activations. As shown in the visualization below, the internal activations for specific phonetic units cluster much more tightly after ASAO normalization, regardless of the original speaker.

LDA projection of normalized activations showing tighter clustering of phonetic features.

Results and Conclusion

ASAO demonstrated robust performance across metrics:

  • Word Error Rate (WER): In the 150-speaker condition, ASAO reduced the WER from 21.8% to 19.8%, a significant 2% absolute reduction.
  • Comparison to Adversarial Training: ASAO outperformed speaker adversarial training frameworks. Proactively subtracting learned offsets proved more effective than the "gradient reversal" approach used in adversarial models.
  • Accuracy: In the 600-speaker condition, senone classification accuracy improved from 39.7% to 41.5%.

While computationally intensive due to the calculation of (speaker, senone)-level means, ASAO represents a powerful step forward in unsupervised, model-internal speaker adaptation for sequence-based neural networks.

Summary updated

Abstract

In this work, we present an unsupervised long short-term memory (LSTM) layer normalization technique that we call adaptation by speaker aware offsets (ASAO). These offsets are learned using an auxiliary network attached to the main senone classifier. The auxiliary network takes main network LSTM activations as input and tries to reconstruct speaker, (speaker,phone) and (speaker,senone)-level averages of the activations by minimizing the mean-squared error. Once the auxiliary network is jointly trained with the main network, during test time we do not need additional information for the test data as the network will generate the offset itself. Unlike many speaker adaptation studies which only adapt fully connected layers, our method is applicable to LSTM layers in addition to fully-connected layers. In our experiments, we investigate the effect of ASAO of LSTM layers at different depths. We also show its performance when the inputs are already speaker adapted by feature space maximum likelihood linear regression (fMLLR). In addition, we compare ASAO with a speaker adversarial training framework. ASAO achieves higher senone classification accuracy and lower word error rate (WER) than both the unadapted models and the adversarial model on the HUB4 dataset, with an absolute WER reduction of up to 2%.
BibTeX
@inproceedings{sari2019learning,
  author = {Leda Sari and Samuel Thomas and Mark A. Hasegawa-Johnson},
  title = {{Learning Speaker Aware Offsets for Speaker Adaptation of Neural Networks}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=7xMn3i1fxXt},
}
Figure