Abstract
In this work, we present an unsupervised long short-term memory (LSTM) layer normalization technique called Adaptation by Speaker Aware Offsets (ASAO). These offsets are learned using an auxiliary network attached to the main senone classifier. The auxiliary network takes main network LSTM activations as input and reconstructs speaker, (speaker, phone), and (speaker, senone)-level averages of the activations by minimizing the mean-squared error. Once jointly trained, the network generates the offset itself during test time without requiring additional speaker information. Unlike many speaker adaptation studies that only target fully connected layers, ASAO is applicable to LSTM layers. Experiments on the HUB4 dataset show that ASAO achieves higher senone classification accuracy and lower word error rate (WER) than unadapted and adversarial models, with an absolute WER reduction of up to 2%.
Introduction: The Challenge of Speaker Variability
In automatic speech recognition (ASR), speaker variability—differences in accent, pitch, and vocal tract shape—creates a significant "mismatch" problem. A model trained on a general population often falters when encountering a new speaker. Traditional solutions typically involve feature-level normalization (like fMLLR) or supervised model-level adaptation. However, there is a growing need for unsupervised methods that can adapt complex sequence models like LSTMs on the fly.
Methodology: The ASAO Framework
The core innovation of this research is the ASAO framework, which introduces an auxiliary network to neutralize speaker-specific variability within the hidden layers of the main classifier.
How ASAO Works
- The Main Network: A standard acoustic state (senone) classifier utilizing LSTM and fully connected (FC) layers.
- The Auxiliary Network: This component "piggybacks" on the main network, taking hidden activations ($h$) from an LSTM layer as input.
- Targeting the Shift: During training, the auxiliary network is tasked with predicting the deviation of a speaker's average activation from the global average across speaker, phone, and senone levels.
- Normalization: The auxiliary network produces a bottleneck feature ($z$), which is transformed by a matrix ($T$) into a speaker-aware offset. This offset is subtracted from the main network’s activations: $\hat{h} = h - T(z)$. This process effectively "strips away" speaker-specific components, leaving a speaker-invariant signal for subsequent layers.
Experimental Setup
The researchers evaluated ASAO using the HUB4 Broadcast News dataset. They tested two primary conditions: a 150-speaker set (more data per speaker) and a 600-speaker set (less data per speaker). Evaluation was performed on a held-out set of speakers to ensure the adaptation was truly unsupervised and capable of generalizing to entirely new voices.
Key Insights and Visualization
Early Adaptation and Complementarity
The study found that adapting the first LSTM layer yielded the most significant gains. As signals move deeper into a network, they naturally become more focused on phonetic content rather than speaker identity. Furthermore, ASAO provided improvements even when input features were already normalized with fMLLR, suggesting it captures internal non-linear variations that linear methods miss.
Visualizing Speaker Invariance
To confirm the effectiveness of the normalization, the authors used Linear Discriminant Analysis (LDA) to project the activations. As shown in the visualization below, the internal activations for specific phonetic units cluster much more tightly after ASAO normalization, regardless of the original speaker.
Results and Conclusion
ASAO demonstrated robust performance across metrics:
- Word Error Rate (WER): In the 150-speaker condition, ASAO reduced the WER from 21.8% to 19.8%, a significant 2% absolute reduction.
- Comparison to Adversarial Training: ASAO outperformed speaker adversarial training frameworks. Proactively subtracting learned offsets proved more effective than the "gradient reversal" approach used in adversarial models.
- Accuracy: In the 600-speaker condition, senone classification accuracy improved from 39.7% to 41.5%.
While computationally intensive due to the calculation of (speaker, senone)-level means, ASAO represents a powerful step forward in unsupervised, model-internal speaker adaptation for sequence-based neural networks.
Summary updated