Introduction
Automatic Speech Recognition (ASR) systems often struggle with "stranger danger"—a significant drop in performance when encountering speakers whose acoustic characteristics differ from the training data. While speaker adaptation techniques like i-vectors are commonly used to mitigate this, they typically require supervised knowledge of speaker identities or auxiliary extraction systems at test time. Furthermore, static utterance-level embeddings fail to account for dynamic changes, such as multiple speakers in a single recording.
Researchers from MERL and UIUC address these challenges by introducing an unsupervised speaker adaptation method inspired by the Neural Turing Machine (NTM). Their approach utilizes an attention-based memory block to store and retrieve speaker characteristics on-the-fly, enabling robust adaptation without external labels.
Proposed Architecture: The Memory Block
The core innovation is the integration of a neural memory block within the ASR encoder. Unlike traditional systems that append a static i-vector to the input, this model "reads" from a memory of training speaker profiles at every frame.
As shown in Figure 1, the process involves:
- Speaker Memory: A fixed matrix storing i-vectors for all speakers in the training set.
- Read Operation: An intermediate encoder layer (Enc1) generates a query $q_t$. An attention mechanism uses this query to retrieve relevant information from the memory.
- M-Vector ($r_t$): The result is a dynamic "M-vector"—a weighted combination of training speaker profiles that best matches the current acoustic frame.
- Integration: The M-vector is concatenated with hidden activations $h_t$ and passed to the subsequent encoder stage (Enc2).
Experimental Setup
The method was evaluated on two benchmarks:
- Wall Street Journal (WSJ): 81 hours of read news.
- TED-LIUM2: 211 hours of TED talks, providing a larger set of speakers for the memory block.
Key Results and Discussion
The M-vector approach demonstrated superior performance across both datasets, often outperforming supervised i-vector baselines.
Performance on WSJ and TED-LIUM2
On the WSJ evaluation set, the M-vector reduced the Word Error Rate (WER) from 5.8% (unadapted) to 4.2%, a 10.6% relative improvement over the supervised i-vector baseline (4.7%). Similar gains were observed on TED-LIUM2, where the M-vector achieved 11.0% WER compared to the 16.7% unadapted baseline.
Optimal Layer for Adaptation
The researchers found that the location of the memory block matters. Experiments on WSJ indicated that injecting speaker information at intermediate layers (e.g., layer 2 or 3) is more effective than at the input layer (layer 0). This suggests that the network benefits from processing raw features into higher-level representations before conditioning them on speaker identity.
Robustness to Speaker Changes
A standout feature of the M-vector is its ability to handle mid-utterance speaker changes. Because the attention mechanism operates at the frame level, it can rapidly update the M-vector as the acoustic profile shifts.
As illustrated in Figure 5, while traditional i-vector systems suffer significantly when speakers change (rising to 15.9% WER on TED-LIUM2), the M-vector system remains much more robust (11.9% WER). In simulated WSJ tests, the M-vector achieved 4.9% WER compared to 7.8% for the standard i-vector system.
Conclusion
By framing speaker adaptation as a dynamic memory-retrieval problem, this work enables E2E ASR systems to adapt to unseen speakers in a completely unsupervised manner. The frame-level attention mechanism not only improves overall accuracy but also provides a unique solution for multi-speaker recordings. Future work may explore "writable" memories to allow the system to learn new speaker representations during inference.
Summary updated