Probabilistic Speaker-Class based Acoustic Modeling for Large Vocabulary Continuous Speech Recognition

Xiangang Li, Dan Su, Zaihu Pang, Xihong WuInterspeech 2012 · OpenReview

Abstract

In this paper, a probabilistic speaker-class (PSC) based acoustic modeling method is proposed for taking into account speaker variability influence in HMM-based speech recognition systems. Firstly, within the context of speaker-class based speech recognition, an experiment is conducted to investigate the performance of speaker-class recognition based on hard-cut speaker clustering. Then, in the proposed method, through introducing the probabilistic latent speaker analysis, the speaker-class dependent acoustic models are trained based on a soft-decision speaker clustering method, and combined by the distribution of speaker-class in the decoding phase. The experiments were conducted on a 600-hour speech corpus, and showed improvement in a large vocabulary continuous speech recognition task.

Introduction

Automatic Speech Recognition (ASR) systems face a persistent challenge: speaker variability. Differences in vocal tract anatomy and speaking styles mean that a single acoustic model often fails to capture the nuances of diverse speakers. To address this, researchers often use speaker-class models—specialized models for groups of similar speakers. However, traditional "hard-cut" clustering, where a speaker is assigned to exactly one class, has significant limitations.

The Problem with Hard-Cut Clustering

The authors identify that the speaker space is often "confused," making rigid classification unreliable. Their initial experiments revealed a "U-curve" in speaker recognition performance. As shown in Figure 3, accuracy is high when there are very few classes (e.g., 4) or when the number of classes approaches the number of individual speakers (e.g., 1024). However, performance drops significantly in the middle range (16 to 128 classes), which is precisely where speaker-class models are typically most useful.

The "U-curve" of speaker-class recognition accuracy as a function of the number of classes.

If a system misclassifies a speaker at the outset, the subsequent recognition performance suffers. This motivates a shift toward a more fluid, probabilistic approach.

Proposed Method: Probabilistic Speaker-Class (PSC) Modeling

The core innovation is the shift from hard classification to soft-decision clustering using Probabilistic Latent Speaker Analysis (PLSA).

  1. Soft Training: Instead of assigning a speaker to one cluster, the system calculates the probability that a speaker belongs to various "latent" speaker classes. Acoustic models are then trained using these weighted contributions.
  2. Latent Speaker Modeling: The system uses a Probabilistic Latent Speaker Model (PLSM) to define relationships between speakers, latent classes, and Gaussian components.
  3. Dynamic Decoding: During decoding, the system maintains a probability distribution across all latent classes at the frame level. It then weights and combines the top $m$ best-matching speaker-class models to calculate the likelihood of the speech.

This allows the system to "blend" models dynamically as the speaker's characteristics are processed, capturing local structures rather than relying on a single global classification.

Experimental Results

The experiments were conducted on a massive 600-hour Mandarin speech corpus, including the 863-1, 863-2, and Intel Continuous Mandarin corpora. The evaluation set consisted of clean utterances from the Hub4 1997 Mandarin broadcast news data.

The PSC method showed clear advantages over both the baseline and traditional hard-clustering methods. The baseline CD-HMM achieved a Character Error Rate (CER) of 24.80%. By contrast, the proposed PSC model with $m=4$ reduced the CER to 22.27%, representing a 10.2% relative reduction in error.

Character Error Rate (CER) improvement as the number of combined speaker-class models (m) increases.

As illustrated in Figure 5, the CER decreases as more speaker-class models are combined, reaching an optimal point around $m=4$ or $5$. Beyond this, the returns diminish while computational costs continue to rise.

Conclusion

The Probabilistic Speaker-Class approach effectively mitigates the risks of speaker misclassification inherent in hard-cut clustering. By leveraging soft-decision clustering and dynamic model blending, the system achieves significant accuracy gains in LVCSR tasks. While the method increases computational requirements for memory and decoding time, the performance improvements justify the overhead, particularly for high-accuracy applications. Future work may focus on optimizing the efficiency of the decoding process to handle real-world, noisy environments.

Summary updated

Abstract

In this paper, a probabilistic speaker-class (PSC) based acoustic modeling method is proposed for taking into account speaker variability influence in HMM-based speech recognition systems. Firstly, within the context of speaker-class based speech recognition, an experiment is conducted to investigate the performance of speaker-class recognition based on hard-cut speaker clustering. Then, in the proposed method, through introducing the probabilistic latent speaker analysis, the speaker-class dependent acoustic models are trained based on a softdecision speaker clustering method, and combined by the distribution of speaker-class in the decoding phase. The experiments were conducted on a 600-hour speech corpus, and showed improvement in a large vocabulary continuous speech recognition task.
BibTeX
@inproceedings{li2012probabilistic,
  author = {Xiangang Li and Dan Su and Zaihu Pang and Xihong Wu},
  title = {{Probabilistic Speaker-Class based Acoustic Modeling for Large Vocabulary Continuous Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=Wc5xBJ8CRD},
}
Figure