Abstract: We present SVCnet, a system for modelling speaker variability. Encoder Neural Networks specialized for each speech sound produce low dimensionality models of acoustical variation, and these models are further combined into an overall model of voice variability. A training procedure is described which minimizes the dependence of this model on which sounds have been uttered. Using the trained model (SVCnet) and a brief, unconstrained sample of a new speaker's voice, the system produces a Speaker Voice Code that can be used to adapt a recognition system to the new speaker without retraining. A system which combines SVCnet with an MS-TDNN recognizer is described.
Introduction
In the early 1990s, a significant performance gap existed between speaker-dependent and speaker-independent speech recognition systems. While humans can rapidly adapt to a new person's accent or vocal characteristics after hearing just a few words, machine learning models typically required extensive enrollment data or complex retraining to achieve similar performance. SVCnet was developed to bridge this gap, enabling a system to identify a speaker's position in a multi-dimensional "voice space" using only a tiny, unconstrained sample of speech.
Methodology: The SVCnet Architecture
SVCnet utilizes a hierarchical architecture based on "bottleneck" neural networks (autoencoders) to create a compact Speaker Voice Code (SVC). The process occurs in two distinct stages:
- Phone Pronunciation Codes (PPCs): For specific speech sounds (e.g., the vowel /EH/), specialized encoder networks compress acoustic variation into a low-dimensional code. This PPC captures how a specific speaker's pronunciation of that sound deviates from the average.
- The Speaker Voice Model: A second "pattern completion" network takes the available PPCs and predicts the speaker's entire vocal profile. Since a speaker rarely utters every possible sound in a short sample, this network is trained to "guess" missing pronunciation codes based on the observed ones.
This SVC is then fed as an auxiliary input into a standard Multi-State Time Delay Neural Network (MS-TDNN) recognizer, providing a bias that helps the system interpret incoming audio more accurately for that specific speaker.
Experimental Evaluation
The researchers evaluated SVCnet using two primary datasets:
- RMSpell: A DARPA database of 120 speakers spelling 15 words aloud, used to visualize SVC stability.
- French Digits: A dataset of over 6,800 telephone-quality digits. Each speaker uttered roughly 9 out of 10 possible digits, testing the system's ability to generalize from incomplete data.
Key Results and Insights
The system demonstrated that the most innovative aspect of SVCnet is its ability to perform pattern completion. By hearing how a user says a few sounds, it can infer their likely pronunciation of others.
- Rapid Stability: As shown in the trajectories above, the SVC reaches a stable point in "speaker space" after only 3 to 4 words. This mimics human-like rapid adaptation.
- Error Reduction: On the French digit task, the baseline error rate of 1.77% was reduced to 0.99% when the SVC was applied across all levels of the recognition network—a 44% relative error reduction.
- Content Independence: The generated SVC proved to be largely independent of the specific words spoken; different sets of words from the same speaker produced highly consistent codes.
Conclusion and Future Work
SVCnet successfully demonstrates that neural networks can perform rapid speaker adaptation by mapping voices into a low-dimensional space. While effective for digits and spelling, future work is needed to scale this approach to more complex vocabularies and to investigate whether the "speaker space" mapped by the network aligns with human perceptual categories like pitch or accent. Additionally, further refinement could reduce within-speaker variation to ensure even greater consistency across different sessions.
Summary updated