Acoustic modeling for spoken dialogue systems based on unsupervised utterance-based selective training

Tobias Cincarek, Tomoki Toda, Hiroshi Saruwatari, Kiyohiro ShikanoInterspeech 2006

TL;DR

Building high-performance acoustic models for speech recognition is expensive due to the need for large, transcribed datasets. This paper introduces an unsupervised method to automatically select the most relevant training utterances from a large, untranscribed pool, significantly reducing transcription costs without sacrificing accuracy. This approach enables more efficient adaptation of acoustic models to specific user groups, like children or adults.

What Problem Does This Paper Address?

Creating robust acoustic models for speech recognition systems, especially for interactive dialogue systems, is a major bottleneck. The primary challenge lies in the substantial cost and time required to collect and meticulously transcribe vast amounts of speech data. This data needs to be representative of various factors like speaker characteristics (age, accent), speaking style, and acoustic conditions. Consequently, a single acoustic model often struggles to perform well across different domains or user groups. While existing methods like active learning and unsupervised learning aim to reduce transcription effort, they often have limitations in how they select data. This paper tackles the problem of cost-effectively building high-performance acoustic models by proposing a novel unsupervised selective training approach that intelligently prunes large untranscribed datasets.

Methods

The core of this paper is the unsupervised utterance-based selective training method. The fundamental idea is to select a subset of untranscribed speech data from a large pool such that the likelihood of a small, manually transcribed development dataset is maximized. This effectively means choosing utterances that are acoustically similar to the target speech.

The process involves:

  1. Initial Data Collection and Labeling: A small portion of the collected speech data is manually transcribed and used to create a development set (e.g., for adult and child speech).
  2. Automatic Transcription of Data Pool: The remaining large pool of untranscribed data is automatically transcribed using an existing speech recognizer.
  3. Utterance-based Selection: A greedy heuristic algorithm called ST_DelScan is employed. This algorithm iteratively examines utterances in the training data pool. An utterance is discarded if its removal leads to an increase in the likelihood of the development data. This process is computationally efficient, allowing for rapid likelihood calculations using HMM sufficient statistics.
  4. Model Retraining: The acoustic model is then retrained using only the selected subset of data.

This method allows for automatic task-adaptation, for instance, to build separate models for adult and child speech, by tailoring the selection process based on the corresponding development data.

Datasets

  • Takemaru Database: Spontaneous Japanese speech data collected from an animated character-based dialogue system installed in a public place in Nara, Japan. This database contains over 500,000 speech and noise inputs collected since November 2002. The data is classified into speaker groups: infants, elementary school children, junior-high school children, adults, and elderly persons.
  • Japanese Newspaper Article Sentences (JNAS) database: Used to build the initial acoustic model for the speech recognizer.
  • Evaluation Data: Randomly selected from data collected over two years, ensuring representation from different speakers and age groups.
  • Development Data: Manually transcribed data, used to guide the selection process. For experiments, this included 2,000 children utterances and 1,000 adult utterances.
  • Data Pool: Consists of all collected data with a duration of 0.7 seconds or more, excluding initial data used for development and evaluation. This pool is automatically transcribed.

Key Insights and Ideas

  • Selective Training is Key: The paper strongly suggests that using all available untranscribed data for training doesn't necessarily lead to the best performance, especially when the data is imbalanced (e.g., more child speech than adult speech). Intelligent selection is crucial.
  • Acoustic Similarity Drives Performance: The core insight is that selecting utterances acoustically similar to a small, representative development set maximizes the likelihood of that development set, leading to better acoustic models.
  • Cost Reduction Without Compromise: The proposed method demonstrates that significant reductions in transcription costs are achievable (potentially cutting costs by half) without a corresponding drop in recognition performance.
  • Automatic Adaptation: The utterance-based selection mechanism naturally facilitates automatic adaptation to specific speaker groups (like children vs. adults) by using group-specific development data.
  • Noise and Irrelevant Data Discarding: The selection process effectively discards noisy or irrelevant utterances, which can otherwise negatively impact model training.

Key Results

  • Adult Acoustic Model: Using the proposed unsupervised selective training on a 200k data pool, word accuracy improved from 75.2% (initial model) to 77.4%, outperforming training on all 200k data (74.2%).
  • Child Acoustic Model: For the child model, word accuracy increased from 58.3% (initial model) to 70.3% when using all pool data, and further improved to 72.5% with unsupervised selective training.
  • Human Transcription Reduction: When using the selected data for supervised EM training (Uns.Sel.+Trans.), a performance comparable to training on all transcribed data was achieved, with only about half of the data needing transcription.
  • Data Pool Selection Ratios: The algorithm selected 8.1% of the data pool for the adult model and 20.0% for the children model.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that supervised MLLR+MAP adaptation, while not outperforming the proposed method, achieved comparable results (77.2% for adult, 72.0% for child).
  • The effectiveness of the ST_DelScan algorithm relies on the quality and representativeness of the initial manually transcribed development data.
  • Future Work:
    • The authors suggest that combining their unsupervised selective training approach with supervised MLLR+MAP adaptation might lead to further improvements.
  • The method's applicability to different scenarios and its potential for further optimization are implied.

Summary updated

BibTeX
@inproceedings{cincarek2006acoustic,
  author = {Tobias Cincarek and Tomoki Toda and Hiroshi Saruwatari and Kiyohiro Shikano},
  title = {{Acoustic modeling for spoken dialogue systems based on unsupervised utterance-based selective training}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure