Speaking style adaptation for spontaneous speech recognition using multiple-regression HMM

Yusuke Ijima, Takeshi Matsubara, Takashi Nose, Takao KobayashiInterspeech 2009

TL;DR

This paper introduces a novel method for adapting spontaneous speech recognition models to different speaking styles (like reading vs. academic presentations) on-the-fly. By using a Multiple-Regression HMM (MRHMM) that models style variations, they achieve improved recognition accuracy, especially when adaptation data is scarce, and can even infer the speaking style of the input.

What Problem Does This Paper Address?

Spontaneous speech, unlike read speech, is characterized by a wide range of acoustic variations due to factors like speaking style, emotional state, and disfluencies. These variations create a mismatch between acoustic models trained on clean, controlled data and real-world spontaneous speech, leading to significant drops in recognition accuracy. While adapting models to new conditions is crucial, doing so in real-time with limited data is a major challenge. This paper specifically tackles the problem of adapting speech recognition models to different speaking styles (reading vs. academic presentation) in a spontaneous speech context, aiming for rapid adaptation with minimal data.

Methods

The core of the proposed technique lies in the Multiple-Regression Hidden Markov Model (MRHMM). Unlike standard HMMs where mean vectors of probability density functions (PDFs) are fixed, in MRHMMs, these mean vectors are modeled as a function of a low-dimensional "style vector." This style vector captures the intensity of a particular speaking style.

The adaptation process involves two key steps:

  1. Style Estimation: For an incoming utterance, the system first estimates the "style vector" using a technique based on maximum likelihood. This is done on a per-utterance basis.
  2. On-line Model Adaptation: Once the style vector is estimated, the MRHMM is used to compute new mean vectors for the PDFs. This effectively adapts the acoustic model to the estimated speaking style of the current utterance.

For training the MRHMM itself, especially when target speaker data is limited, they employ a strategy that starts with a speaker-independent (SI) model trained on a large corpus of read speech. This SI model is then adapted to the target speaker's specific styles using techniques like MLLR and MAP. Finally, the MRHMM is trained using these style-adapted HMMs and corresponding style vectors.

Datasets

  • Corpus of Spontaneous Japanese (CSJ): This dataset was used for evaluating the proposed technique. It contains spontaneous speech from various speakers, including reading and academic presentation (AP) styles.
  • For MRHMM training, 15 utterances per style per target speaker were used.
  • For testing, 100 utterances per style per target speaker were used.
  • Japanese Newspaper Article Sentences (JNAS): This dataset was used to train the initial speaker-independent (SI) reading style model. It contains read speech data from 209 speakers.

Key Insights and Ideas

  • Style as a Controllable Parameter: The paper frames speaking style not just as a nuisance variable but as a parameter that can be explicitly modeled and controlled within the acoustic model.
  • Rapid Adaptation with Minimal Data: The MRHMM approach allows for effective adaptation even with very small amounts of data (e.g., 5 utterances for training, 1 utterance for adaptation), which is crucial for real-world scenarios.
  • Dual Information Extraction: The technique not only improves speech recognition accuracy but also provides "paralinguistic information" – the estimated speaking style – as a byproduct. This is a significant advantage over traditional methods.
  • Bridging SI and Target Models: The strategy of using an SI model as a starting point and then adapting it to specific styles and speakers is effective for overcoming data scarcity issues.

Key Results

  • Improved Recognition Accuracy: The MRHMM-based approach demonstrated superior phoneme recognition error rates compared to standard HMMs adapted from the SI model (both style-dependent and style-independent). For example, in Figure 4, MRHMM achieved error rates around 11.8-13.2% for AP style and 17.9-20.0% for Reading style, outperforming the other methods.
  • Effective Style Classification: The system showed promising results in classifying the speaking style of input utterances. Figure 3 illustrates classification error rates as low as 0-10% for some speakers, demonstrating the effectiveness of the style estimation component. Table 2 shows average correct classification rates of 94.9% to 99.9% for AP style and 96.0% to 99.2% for Reading style across different speakers.

Limitations and Future Work

  • Limited Style Space: The current work focuses on a one-dimensional style space to represent two distinct styles (reading and AP). The authors suggest exploring the effectiveness of the proposed technique using other speaking styles in the future.
  • Covariance Parameters: The adaptation did not involve adapting the covariance parameters of the PDFs, which might offer further improvements.
  • Phoneme Label Dependency: The style estimation process requires a phoneme label sequence of the input utterance, which is obtained through a two-pass recognition process. Investigating methods that reduce this dependency could be beneficial.

Summary updated

BibTeX
@inproceedings{ijima2009speaking,
  author = {Yusuke Ijima and Takeshi Matsubara and Takashi Nose and Takao Kobayashi},
  title = {{Speaking style adaptation for spontaneous speech recognition using multiple-regression HMM}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure