Rapid Connectionist Speaker Adaptation

Michael Witbrock, Patrick Haffner

Abstract: We present SVCnet, a system for modelling speaker variability. Encoder Neural Networks specialized for each speech sound produce low dimensionality models of acoustical variation, and these models are further combined into an overall model of voice variability. A training procedure is described which minimizes the dependence of this model on which sounds have been uttered. Using the trained model (SVCnet) and a brief, unconstrained sample of a new speaker's voice, the system produces a Speaker Voice Code that can be used to adapt a recognition system to the new speaker without retraining. A system which combines SVCnet with an MS-TDNN recognizer is described.

Introduction

In the early 1990s, a significant performance gap existed between speaker-dependent and speaker-independent speech recognition systems. While humans can rapidly adapt to a new person's accent or vocal characteristics after hearing just a few words, machine learning models typically required extensive enrollment data or complex retraining to achieve similar performance. SVCnet was developed to bridge this gap, enabling a system to identify a speaker's position in a multi-dimensional "voice space" using only a tiny, unconstrained sample of speech.

Methodology: The SVCnet Architecture

SVCnet utilizes a hierarchical architecture based on "bottleneck" neural networks (autoencoders) to create a compact Speaker Voice Code (SVC). The process occurs in two distinct stages:

  1. Phone Pronunciation Codes (PPCs): For specific speech sounds (e.g., the vowel /EH/), specialized encoder networks compress acoustic variation into a low-dimensional code. This PPC captures how a specific speaker's pronunciation of that sound deviates from the average.
  2. The Speaker Voice Model: A second "pattern completion" network takes the available PPCs and predicts the speaker's entire vocal profile. Since a speaker rarely utters every possible sound in a short sample, this network is trained to "guess" missing pronunciation codes based on the observed ones.
The overall SVCnet architecture, showing the flow from speech input to the generation of the Speaker Voice Code (SVC).

This SVC is then fed as an auxiliary input into a standard Multi-State Time Delay Neural Network (MS-TDNN) recognizer, providing a bias that helps the system interpret incoming audio more accurately for that specific speaker.

Experimental Evaluation

The researchers evaluated SVCnet using two primary datasets:

  • RMSpell: A DARPA database of 120 speakers spelling 15 words aloud, used to visualize SVC stability.
  • French Digits: A dataset of over 6,800 telephone-quality digits. Each speaker uttered roughly 9 out of 10 possible digits, testing the system's ability to generalize from incomplete data.

Key Results and Insights

The system demonstrated that the most innovative aspect of SVCnet is its ability to perform pattern completion. By hearing how a user says a few sounds, it can infer their likely pronunciation of others.

SVC trajectories for four different speakers, showing how the voice code stabilizes rapidly in the speaker space.
  • Rapid Stability: As shown in the trajectories above, the SVC reaches a stable point in "speaker space" after only 3 to 4 words. This mimics human-like rapid adaptation.
  • Error Reduction: On the French digit task, the baseline error rate of 1.77% was reduced to 0.99% when the SVC was applied across all levels of the recognition network—a 44% relative error reduction.
  • Content Independence: The generated SVC proved to be largely independent of the specific words spoken; different sets of words from the same speaker produced highly consistent codes.

Conclusion and Future Work

SVCnet successfully demonstrates that neural networks can perform rapid speaker adaptation by mapping voices into a low-dimensional space. While effective for digits and spelling, future work is needed to scale this approach to more complex vocabularies and to investigate whether the "speaker space" mapped by the network aligns with human perceptual categories like pitch or accent. Additionally, further refinement could reduce within-speaker variation to ensure even greater consistency across different sessions.

Summary updated

Abstract

We present SVCnet, a system for modelling speaker variability. Encoder Neural Networks specialized for each speech sound produce low dimensionality models of acoustical variation, and these models are further combined into an overall model of voice variability. A training procedure is described which minimizes the dependence of this model on which sounds have been uttered. Using the trained model (SVCnet) and a brief, unconstrained sample of a new speaker's voice, the system produces a Speaker Voice Code that can be used to adapt a recognition system to the new speaker without retraining. A system which combines SVCnet with an MS-TDNN recognizer is described
BibTeX
@inproceedings{witbrock1992rapid,
  author = {Michael Witbrock and Patrick Haffner},
  title = {{Rapid Connectionist Speaker Adaptation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {1992},
  url = {https://openreview.net/forum?id=UWHRF1-kYh},
}
Figure