Semi-automated Speaker Adaptation: How to Control the Quality of Adaptation?

Andrey V. Savchenko

Abstract

This paper introduces a "human-in-the-loop" approach to speaker adaptation in automatic speech recognition (ASR). By allowing users to manually verify synthesized versions of their own voice and employing automated stability checks, the system creates a high-quality, personalized phonetic database. This method significantly outperforms traditional "black-box" adaptation techniques like MAP or MLLR, reducing vowel recognition error rates from 8.1% to 4.4% and word recognition errors in specific vocabularies from 9.2% to 0.9%.

Introduction

Most modern ASR systems utilize Speaker-Independent (SI) models trained on diverse datasets to ensure broad usability. However, achieving high accuracy for a specific individual often requires Speaker-Dependent (SD) adaptation. Traditional adaptation methods face two primary challenges: high data requirements (often 10–30 minutes of speech) and the "black box" problem, where users lack feedback on whether the system has accurately captured their unique vocal characteristics. This research proposes a semi-automated procedure focused on Russian vowels to give users direct control over the adaptation quality.

Methods

The proposed workflow consists of two primary stages: a verification loop for database construction and a classifier fusion mechanism for recognition.

The Verification Loop

To ensure the integrity of the speaker's phonetic profile, the system implements a feedback loop:

  1. LPC Synthesis: Upon pronouncing a syllable, the system extracts the vowel and uses Linear Prediction Coding (LPC) coefficients to synthesize the sound.
  2. User Verification: The synthesized sound is played back to the user. If the user determines the synthesis does not accurately reflect their original pronunciation, the recording is repeated.
  3. Stability Check: The user repeats the vowel, and the system calculates the Itakura-Saito (IS) divergence between the two instances. High divergence triggers a repeat, filtering out inconsistent pronunciations.
Flowchart of the proposed semi-automated speaker adaptation and recognition system.

Classifier Fusion

The system combines the strengths of general and personalized models. A global SI model (CMU Sphinx) provides initial probabilities, which are then weighted and fused with outputs from a custom SD classifier. The SD classifier compares input frames to the verified user database using Mel-frequency cepstral coefficients (MFCC) and Euclidean distance.

Experimental Setup and Results

The approach was evaluated using 10 speakers and two specific Russian vocabularies: drug names (1,913 entries) and city names (1,830 entries). Recordings were made at 8000 Hz with a 30 dB Signal-to-Noise Ratio (SNR).

The semi-automated approach proved highly efficient, requiring an average of only 2.75 minutes per speaker for setup. In clean conditions, word recognition error for the cities vocabulary dropped from 9.2% (SI) to 0.9% (Proposed). Furthermore, the method demonstrated superior performance compared to standard MAP/MLLR adaptation.

Error rates for Russian vowel recognition across different experimental conditions (a-d).

The fusion method also showed robustness to noise. Even when artificial white noise reduced the SNR to 10 dB, the proposed system maintained significantly higher accuracy than standard models.

Conclusion

By integrating human verification and automated stability checks, this semi-automated adaptation method provides a fast, reliable, and transparent way to improve ASR performance. While currently optimized for isolated syllable recognition in Russian, future work will explore extending this logic to continuous speech and other languages.

Summary updated

Abstract

Since the early 1990s, speaker adaptation have become one of the intensive areas in speech recognition. State-of-the-art batch-mode adaptation algorithms assume that speech of particular speaker contains enough information about the user’s voice. In this article we propose to allow the user to manually verify if the adaptation is useful. Our procedure requires the speaker to pronounce syllables containing each vowel of particular language. The algorithm contains two steps looping through all syllables. At first, LPC analysis is performed for extracted vowel and the LPC coefficients are used to synthesize the new sound (with a fixed pitch period) and play it. If this synthesized sound is not perceived by the user as an original one then the syllable should be recorded again. At the second stage, speaker is asked to produce another syllable with the same vowel to automatically verify the stability of pronunciation. If two signals are closed (in terms of the Itakura-Saito divergence) then the sounds are marked as ”good” for adaptation. Otherwise both steps are repeated. In the experiment we examine a problem of vowel recognition for Russian language in our voice control system which fuses two classifiers: the CMU Sphinx with speaker-independent acoustic model and Euclidean comparison of MFCC features of model vowel and input signal frames. Our results support the statement that the proposed approach provides better accuracy and reliability in comparison with traditional MAP/MLLR techniques implemented in the CMU Sphinx.
BibTeX
@inproceedings{savchenko2014semi,
  author = {Andrey V. Savchenko},
  title = {{Semi-automated Speaker Adaptation: How to Control the Quality of Adaptation?}},
  booktitle = {ICISP},
  year = {2014},
  doi = {10.1007/978-3-319-07998-1\_73},
}
Figure