Unsupervised discovery and training of maximally dissimilar cluster models

Françoise Beaufays, Vincent Vanhoucke, Brian StropeInterspeech 2010 · OpenReview

Automatic Speech Recognition (ASR) systems face a significant challenge in acoustic variability. Users interacting with services like Google Voice Search do so in diverse environments—from quiet offices to windy streets—using a wide array of vocal characteristics. This paper, presented at Interspeech 2010, introduces a fully unsupervised method to cluster massive amounts of speech data into "maximally dissimilar" groups based on these acoustic conditions, enabling the training of specialized models that outperform universal baselines.

The Challenge of Short-Form Queries

Acoustic modeling typically involves a trade-off between universal models, which provide broad but mediocre coverage, and speaker-dependent models, which are highly accurate but require substantial data from a single individual. For short-form voice queries, traditional adaptation techniques often fail because the audio snippets are too brief to estimate powerful speaker transforms. Furthermore, background noise and channel distortion are often more significant than the speaker's identity. The researchers sought a way to build specialized models for these diverse conditions without requiring human-labeled metadata for millions of hours of audio.

Hierarchical Clustering Methodology

The proposed approach uses a hierarchical, tree-based method to discover acoustic factors directly from untranscribed data. The process involves four key steps:

  1. Acoustic Signatures (Supervectors): Each utterance is converted into a "supervector" of Gaussian posteriors, summarizing acoustic variability in a compact format.
  2. Distributed PCA: To process tens of millions of utterances, the team implemented an iterative Principal Component Analysis (PCA) using MapReduce to identify dominant axes of variability.
  3. Clustering Tree: Using Vector Quantization (VQ), the data is iteratively split into a binary tree. Each node represents a cluster of acoustically similar utterances.
  4. Unsupervised BMMI Training: Specialized models are refined for each cluster using Boosted Maximum Mutual Information (BMMI) training, utilizing the system's own transcript hypotheses for untranscribed data.
Hierarchical clustering tree showing performance gains (WER/SER) and relative improvements across nodes.

Latent Factor Discovery

While the algorithm is entirely unsupervised, the resulting clusters represent physically meaningful acoustic factors. The initial splits in the tree often separate speakers by gender, while subsequent splits differentiate between "clean" and "noisy" speech.

A key finding was that "loudness" (specifically the 95th percentile of filterbank energy) served as a major differentiator. Users naturally speak louder in high-noise environments, and the algorithm successfully grouped these challenging utterances together. This allows the system to "grow" by adding specialized models for specific acoustic niches rather than simply adding more data to a single global model.

Loudness distribution for clusters N010 and N011, illustrating how the unsupervised method separates acoustic conditions.

Key Results and Performance

The cluster-based ensemble achieved a relative reduction in Sentence Error Rate (SER) of approximately 5% across the entire test set. However, the gains were even more pronounced in difficult acoustic conditions. For instance, cluster $N_{0100}$ saw a relative SER reduction of nearly 11%.

The researchers noted that "staged" training—where a child node's model is adapted from its parent rather than the root—was most effective, as it allowed models to inherit general knowledge while specializing. While the system demonstrated high scalability, the authors noted that serving hundreds of specialized models in real-time remains an engineering challenge, suggesting that production systems might prioritize only the most effective clusters for common "difficult" conditions.

Summary updated

BibTeX
@inproceedings{beaufays2010unsupervised,
  author = {Françoise Beaufays and Vincent Vanhoucke and Brian Strope},
  title = {{Unsupervised discovery and training of maximally dissimilar cluster models}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=6bx5JwMnTMJ},
}
Figure