Speech data clustering based on phoneme error trend for unsupervised acoustic model adaptation

Taichi Asami, Satoshi Kobashikawa, Hirokazu Masataki, Osamu Yoshioka, Satoshi TakahashiInterspeech 2012

TL;DR

This paper introduces a novel method for adapting acoustic models in speech recognition by clustering adaptation data based on phoneme error trends, rather than traditional acoustic similarity. This approach leads to more robust adaptation, especially for unknown speakers, by using compact "phoneme error trend vectors."

What Problem Does This Paper Address?

Speech recognition systems, particularly those used in call centers, collect vast amounts of speech data from diverse, unknown speakers. To improve recognition accuracy, unsupervised acoustic model adaptation using this data is promising. A key challenge is effectively clustering this adaptation data to create specialized acoustic models. Traditional methods rely on acoustic similarity between speech samples, often using Gaussian Mixture Models (GMMs). While effective for known speakers, this approach struggles with unknown speakers because the acoustic space is vast and difficult to cover comprehensively. The authors propose a new clustering strategy that focuses on how a speech sample's acoustic model performs in terms of phoneme discrimination, hypothesizing that this "error trend" is a more compact and informative characteristic for clustering.

Methods

The core of the proposed method is the extraction of a "phoneme error trend vector" for each speech sample. This vector captures the deviation of phoneme posterior probabilities for a given sample from the average probabilities across all adaptation data.

  1. Phoneme Error Trend Vector Extraction:
    • For each speech sample $U_i$, the phoneme posterior probability vector $P_i$ is calculated. This is done by averaging frame-level posterior probabilities where a specific phoneme is the best hypothesis.
  2. A global average posterior probability vector, $P_{all}$, is computed from the concatenation of all adaptation samples.
  3. The phoneme error trend vector $E_i$ is then calculated as the difference: $E_i = P_i - P_{all}$. This vector highlights phonemes for which the model's discrimination is unusually high or low for that specific sample.
  4. The dimension of this vector is at most the number of phonemes in the language (e.g., 30 for Japanese).
  5. Clustering and Adaptation:
    • The set of phoneme error trend vectors $\{E_1, E_2, \dots, E_N\}$ is clustered using standard vector clustering algorithms (k-means in this paper).
  6. Cosine similarity is used as the distance metric for clustering.
  7. For each cluster, a cluster-adapted acoustic model is trained using unsupervised MAP adaptation from a base acoustic model.
  8. Model Selection for Recognition:
    • When a new recognition target sample arrives, its phoneme error trend vector $E_r$ is extracted.
  9. The centroid of each cluster is pre-computed.
  10. The cluster whose centroid has the highest cosine similarity with $E_r$ is selected.
  11. The acoustic model corresponding to this selected cluster is used for recognition.

Datasets

  • Evaluation Task Data: 391 phone calls (30 hours) of spontaneous conversation from 24 speakers. These calls were recorded at 16 kHz / 16 bit. This data was used for both adaptation and evaluation.
  • Base Acoustic Model Training Data: A 224-hour training set was used to train the speaker-independent (SI) 3-state left-to-right HMMs using dMMI criterion.
  • Language Model Training Data: A set of 1 million words from manual transcripts of call center recordings.

The paper does not introduce new benchmark datasets but uses a substantial, realistic dataset representative of call center scenarios.

Key Insights and Ideas

  • Phoneme Error Trends as a Clustering Feature: The central idea is that the pattern of errors a speech recognition model makes on a particular utterance is a more robust and compact feature for adaptation clustering than raw acoustic similarity. This shifts focus from "what the speech sounds like" to "how the model struggles/succeeds with this speech."
  • Compactness and Robustness: The phoneme error trend vectors are significantly lower-dimensional (at most the number of phonemes) compared to GMM-based representations. This compactness is hypothesized to provide better robustness against unseen data, as it's easier to cover a smaller feature space.
  • Unsupervised Adaptation Effectiveness: The paper demonstrates that unsupervised adaptation, when guided by this novel clustering method, can significantly improve performance over a baseline model.
  • Speaker Open vs. Closed Test Significance: The proposed method shows comparable performance to traditional speaker clustering in "speaker closed" tests (where adaptation and test speakers overlap) but offers a significant advantage in "speaker open" tests (where test speakers are unseen during adaptation). This highlights its strength in handling novel speakers.

Key Results

  • Speaker Closed Test: The proposed "Adapted-CD-Proposed" method achieved performance comparable to the traditional "Adapted-CD-CLR" (speaker clustering) method, with a character correctness of 84.55% vs. 84.53%. This indicates the phoneme error trend vectors are sufficiently expressive for known speakers.
  • Speaker Open Test: The proposed method ("Adapted-CD-Proposed") significantly outperformed the traditional speaker clustering method ("Adapted-CD-CLR") in speaker open tests. Character correctness was 82.28% for the proposed method versus 82.04% for CD-CLR. This difference was statistically significant (p < 0.01).
  • Overall Performance: Across both closed and open tests, the proposed method consistently achieved the best or near-best performance, with an average character correctness of 83.42% compared to 83.29% for CD-CLR.
  • Robustness: The performance degradation from the generic "Adapted-SI" (single adapted model) was smaller for the proposed method in the speaker open test, indicating better robustness to unknown speakers.

Limitations and Future Work

  • Computational Cost of Model Selection: The paper notes that a significant portion of the computation time for model selection is spent on the monophone typewriter procedure to extract the phoneme error trend vector for the target sample. While this is still shorter than the overall recognition time, it's a point for optimization.
  • Optimal Number of Clusters: While the optimal number of clusters (48) was determined for the experiments, finding this optimal number in a fully unsupervised manner could be a direction for future work.
  • Generalizability: While the paper demonstrates effectiveness on call center data, further validation on other types of speech data (e.g., broadcast news, different acoustic environments) would be beneficial.
  • Exploration of Other Clustering Algorithms: The paper used k-means. Investigating other clustering algorithms with the phoneme error trend vectors could yield further improvements.

Summary updated

BibTeX
@inproceedings{asami2012speech,
  author = {Taichi Asami and Satoshi Kobashikawa and Hirokazu Masataki and Osamu Yoshioka and Satoshi Takahashi},
  title = {{Speech data clustering based on phoneme error trend for unsupervised acoustic model adaptation}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure