Acoustic model adaptation for coded speech using synthetic speech

Koji Tanaka, Fuji Ren, Shingo Kuroiwa, Satoru Tsuge

Introduction

Maintaining high-quality speech recognition in Voice over IP (VoIP) and cellular networks is a persistent challenge. Audio is typically compressed using codecs to save bandwidth, which introduces spectral artifacts and information loss. While humans can often ignore these distortions, standard Automatic Speech Recognition (ASR) systems—usually trained on clean, uncompressed audio—suffer significant performance degradation when encountering coded speech.

The Challenge of Coded Speech

The primary issue is the acoustic mismatch between the features expected by the Hidden Markov Model (HMM) and the features received from a codec. Traditionally, mitigating this requires "multi-condition training," which involves collecting and labeling thousands of utterances for every specific codec and bitrate. This process is both expensive and time-consuming.

Impact of speech coding on word accuracy for baseline and coded-speech models.

As shown in the figure above, the baseline HMM's accuracy drops sharply as the bitrate decreases (e.g., from clean speech to G.723.1 at 5.3 kbps). A model trained directly on coded speech (Coded-Speech HMM) provides an upper bound for performance but requires a dedicated database for each codec.

Proposed Methods

The authors propose a "zero-data" adaptation technique that eliminates the need for real-world recorded samples. The core idea is to use HMM-based speech synthesis as a bridge: instead of using the HMM solely for recognition, it is used to generate speech that represents the model's internal states.

Proposed adaptation framework using HMM-based speech synthesis to simulate codec distortion.

Two primary strategies were evaluated:

  1. Sentence-based Adaptation: An HMM-based synthesizer generates 503 phonetically balanced sentences. These synthetic utterances are passed through the target codec and used to adjust the acoustic model via Maximum Likelihood Linear Regression (MLLR).
  2. Mean-vector based Adaptation: This more direct approach generates short speech segments corresponding to the "mean" of each HMM state. These segments are processed by the codec, and the resulting mel-cepstral coefficients are used to directly update the HMM's mean vectors.

Experimental Evaluation

The study utilized the ASJ-JNAS Japanese database for training (5,168 utterances) and a test set of 100 utterances. The target codec was ITU-T G.723.1 at bitrates of 6.3 kbps and 5.3 kbps.

Results and Discussion

The mean-vector based adaptation emerged as the superior method, outperforming the more complex sentence-based approach. The researchers hypothesize that synthetic sentences may lack the phonetic diversity of real speech (representing only one "average" synthetic speaker), whereas direct mean-vector adaptation precisely targets the model's parameters.

Method 6.3 kbps 5.3 kbps
Baseline (Clean Model) 80.4% 76.1%
Mean-vector Adaptation (Proposed) 82.0% 79.0%
Coded Speech HMM (Upper Limit) 83.0% 80.7%

On the 5.3 kbps codec, the proposed method reduced the relative word error rate by approximately 12%, effectively closing the gap between the baseline and the theoretical upper limit of a model trained on real coded data.

Conclusion

This research demonstrates that an acoustic model can serve as its own source of adaptation data. By simulating codec distortions through synthesis, ASR systems can be rapidly adapted to new telecommunications standards without the need for expensive data collection. Future work will explore adapting covariance matrices and expanding the technique to a broader range of speakers and modern codecs.

Summary updated

Abstract

In this paper, we describe a novel acoustic model adaptation technique which generates "speaker-independent" HMM for the target environment. Recently, personal digital assistants like cellular phones are shifting to IP terminals. The encoding-decoding process utilized for transmitting over IP networks deteriorates the quality of speech data. This deterioration causes degradation in speech recognition performance. Acoustic model adaptations can improve recognition performance. However, the conventional adaptation methods usually require a large amount of adaptation data. The proposed method uses HMM-based speech synthesis to generate adaptation data from the acoustic model of HMM-based speech recognizer, and consequently does not require any speech data for adaptation. Experimental results on G.723.1 coded speech recognition show that the proposed method improves speech recognition performance. A relative word error rate reduction of approximately 12% was observed.
BibTeX
@inproceedings{tanaka2004acoustic,
  author = {Koji Tanaka and Fuji Ren and Shingo Kuroiwa and Satoru Tsuge},
  title = {{Acoustic model adaptation for coded speech using synthetic speech}},
  booktitle = {Interspeech},
  year = {2004},
  doi = {10.21437/interspeech.2004-17},
}
Figure