Abstract
This paper introduces a "human-in-the-loop" approach to speaker adaptation in automatic speech recognition (ASR). By allowing users to manually verify synthesized versions of their own voice and employing automated stability checks, the system creates a high-quality, personalized phonetic database. This method significantly outperforms traditional "black-box" adaptation techniques like MAP or MLLR, reducing vowel recognition error rates from 8.1% to 4.4% and word recognition errors in specific vocabularies from 9.2% to 0.9%.
Introduction
Most modern ASR systems utilize Speaker-Independent (SI) models trained on diverse datasets to ensure broad usability. However, achieving high accuracy for a specific individual often requires Speaker-Dependent (SD) adaptation. Traditional adaptation methods face two primary challenges: high data requirements (often 10–30 minutes of speech) and the "black box" problem, where users lack feedback on whether the system has accurately captured their unique vocal characteristics. This research proposes a semi-automated procedure focused on Russian vowels to give users direct control over the adaptation quality.
Methods
The proposed workflow consists of two primary stages: a verification loop for database construction and a classifier fusion mechanism for recognition.
The Verification Loop
To ensure the integrity of the speaker's phonetic profile, the system implements a feedback loop:
- LPC Synthesis: Upon pronouncing a syllable, the system extracts the vowel and uses Linear Prediction Coding (LPC) coefficients to synthesize the sound.
- User Verification: The synthesized sound is played back to the user. If the user determines the synthesis does not accurately reflect their original pronunciation, the recording is repeated.
- Stability Check: The user repeats the vowel, and the system calculates the Itakura-Saito (IS) divergence between the two instances. High divergence triggers a repeat, filtering out inconsistent pronunciations.
Classifier Fusion
The system combines the strengths of general and personalized models. A global SI model (CMU Sphinx) provides initial probabilities, which are then weighted and fused with outputs from a custom SD classifier. The SD classifier compares input frames to the verified user database using Mel-frequency cepstral coefficients (MFCC) and Euclidean distance.
Experimental Setup and Results
The approach was evaluated using 10 speakers and two specific Russian vocabularies: drug names (1,913 entries) and city names (1,830 entries). Recordings were made at 8000 Hz with a 30 dB Signal-to-Noise Ratio (SNR).
The semi-automated approach proved highly efficient, requiring an average of only 2.75 minutes per speaker for setup. In clean conditions, word recognition error for the cities vocabulary dropped from 9.2% (SI) to 0.9% (Proposed). Furthermore, the method demonstrated superior performance compared to standard MAP/MLLR adaptation.
The fusion method also showed robustness to noise. Even when artificial white noise reduced the SNR to 10 dB, the proposed system maintained significantly higher accuracy than standard models.
Conclusion
By integrating human verification and automated stability checks, this semi-automated adaptation method provides a fast, reliable, and transparent way to improve ASR performance. While currently optimized for isolated syllable recognition in Russian, future work will explore extending this logic to continuous speech and other languages.
Summary updated