The use of sense in unsupervised training of acoustic models for ASR systems

Rita Singh, Benjamin Lambert, Bhiksha RajInterspeech 2010 · DOI

Introduction

Training robust Automatic Speech Recognition (ASR) systems typically requires vast amounts of human-transcribed audio. Unsupervised training offers a potential solution by using an existing "bootstrap" model to transcribe new audio, selecting the most reliable segments, and iteratively retraining. However, this process often faces a "cold start" problem: if the initial model is a poor match for the target audio (e.g., due to different accents or recording conditions), it produces high-error transcriptions. Standard metrics like acoustic likelihood often fail to detect these errors, causing the system to "learn" its own mistakes.

To overcome this, the authors propose a novel Content-Based Selection Metric called "sensicality." This metric provides an external sanity check by judging whether a hypothesized transcription actually makes sense in the real world, independent of the acoustic model's internal logic.

Methodology: The Sensicality Metric

Instead of relying on complex linguistic or grammatical rules, the "sensicality" metric uses a frequentist approach based on the Google N-gram dataset. This dataset contains counts of word sequences (up to 5 words long) found across a trillion words on the web.

The system evaluates hypothesized transcriptions by looking at every sub-sequence of 2 to 5 words. It calculates a score based on how often these patterns appear in the N-gram data, giving higher weight to longer, more frequent patterns. This allows the system to distinguish between coherent speech and the "word salad" often produced by failing ASR models.

Examples of high-scoring (sensical) sentences versus low-scoring sentences that were rejected by the system.

Iterative Training Loop

The sensicality metric is integrated into an iterative training framework. As shown in the process below, an utterance is only used for retraining if it passes two distinct tests: it must meet a minimum threshold for acoustic likelihood (matching the sound) and a threshold for sensicality (matching known linguistic patterns).

The iterative unsupervised training loop, highlighting the dual selection process based on acoustic likelihood and sensicality.

This creates a "hill-climbing" effect. Even if the system only finds a few sensible snippets initially, those high-quality samples improve the acoustic model. The improved model then produces better transcriptions in the next round, which pass the sensicality filter more easily, allowing the training data set to grow organically.

Key Results and Insights

The authors tested this approach by using a bootstrap model trained on the Switchboard corpus to transcribe the Fisher Phase I corpus—a deliberate mismatch designed to simulate a high-error starting point.

  • Baseline Performance: The initial Word Error Rate (WER) was a staggering 88.5%.
  • Likelihood-only vs. Sensicality: A standard approach using only acoustic likelihood improved quickly but soon "saturated" or got stuck at approximately 79.5% WER.
  • Sensicality Gains: By incorporating the sensicality metric, the system continued to improve over seven iterations, eventually reaching a WER of 75.3%.

This research demonstrates that external validation—bringing in massive amounts of outside world knowledge—is more robust for noisy ASR than relying solely on the model's internal confidence. While computationally more expensive than simple scoring, the use of N-grams provides a powerful mechanism for unsupervised learning in challenging, mismatched environments.

Summary updated

BibTeX
@inproceedings{singh2010use,
  author = {Rita Singh and Benjamin Lambert and Bhiksha Raj},
  title = {{The use of sense in unsupervised training of acoustic models for ASR systems}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-23},
}
Figure