Lattice-Based Lightly-Supervised Acoustic Model Training

Joachim Fainberg, Ondrej Klejch, Steve Renals, Peter Bell

Abstract

This paper presents a novel hybrid approach to training Automatic Speech Recognition (ASR) systems by integrating "lightly-supervised" and "semi-supervised" learning. By merging inaccurate transcripts with machine-generated lattices using Finite State Transducers (FSTs), the researchers preserve uncertainty in the training data. This method achieves up to a 20% relative reduction in word error rates (WER) on broadcast data from the Scottish Parliament, demonstrating that even highly imperfect subtitles can significantly enhance model performance when processed as lattices rather than fixed text.

Introduction

High-quality ASR systems typically require vast amounts of audio paired with precise manual transcriptions. Given the prohibitive cost of manual labor, researchers often leverage "lightly-supervised" data, such as closed captions or subtitles. However, these transcripts are frequently paraphrased, incomplete, or misaligned. Conventional techniques attempt to "fix" these transcripts into a single best-guess text or filter out noisy segments. This paper argues that modern discriminative training objectives, such as Lattice-Free MMI, benefit from the probabilistic "wiggle room" provided by lattices—graphs representing multiple possible word sequences. The authors propose a method to utilize messy human transcripts while maintaining this beneficial uncertainty.

Methods

The core contribution is a Lattice Combination algorithm implemented via Finite State Transducers (FSTs). The process involves three primary components: a transcript transducer ($R$) representing the messy human transcript, a hypothesis lattice ($H$) generated by decoding the audio with a biased language model, and an edit transducer ($E$) that allows for insertions, deletions, and substitutions.

These are composed into a single structure: $T = R \circ E \circ H$. The scoring logic prioritizes matches (cost of -1) over other edits (cost of 0), forcing the algorithm to maximize the inclusion of transcript words while filling gaps with machine hypotheses.

Lattice combination process showing how transcripts and hypotheses are merged to preserve uncertainty.

Crucially, the resulting transducer is pruned to retain multiple likely paths, creating a "supervision lattice." This lattice "collapses" into a single path where the transcript and audio align but remains "wide" (uncertain) where they disagree. Additionally, a deletion penalty was introduced to discourage the model from skipping words, a common issue in lightly-supervised training.

Datasets and Evaluation

The models were initially trained on 179 hours of BBC news data from the Multi-Genre Broadcast (MGB) corpus. Adaptation and testing were performed on data from the Scottish Parliament, a challenging "in-the-wild" dataset. The provided subtitles for this set were highly inaccurate, with a 55.3% WER relative to a true reference. Evaluation was conducted on 6.8 hours of audio across 40 speakers.

Key Results

The proposed lattice combination method significantly outperformed standard baselines. While a standard semi-supervised approach reduced the WER from 30.0% to 28.6%, the lattice combination method achieved a WER of 23.6%. Further refinements, including a biased language model and a deletion penalty, brought the WER down to 22.8%.

Relative WER reduction across different methods, highlighting the superiority of combined lattice-based supervision.

These results represent a 20% relative WER reduction compared to the semi-supervised baseline, confirming that preserving uncertainty through lattices is more effective than training on "corrected" single-path text strings.

Conclusion

While effective, the current implementation of the edit transducer $E$ faces computational challenges as its search space grows quadratically with input length. Future work will focus on optimizing this through factored transducers and exploring how these improvements scale with larger datasets. The study highlights that even very messy subtitles contain valuable signals for ASR training if the system is allowed to account for their inherent uncertainty.

Summary updated

Abstract

In the broadcast domain there is an abundance of related text data and partial transcriptions, such as closed captions and subtitles. This text data can be used for lightly supervised training, in which text matching the audio is selected using an existing speech recognition model. Current approaches to light supervision typically filter the data based on matching error rates between the transcriptions and biased decoding hypotheses. In contrast, semi-supervised training does not require matching text data, instead generating a hypothesis using a background language model. State-of-the-art semi-supervised training uses lattice-based supervision with the lattice-free MMI (LF-MMI) objective function. We propose a technique to combine inaccurate transcriptions with the lattices generated for semi-supervised training, thus preserving uncertainty in the lattice where appropriate. We demonstrate that this combined approach reduces the expected error rates over the lattices, and reduces the word error rate (WER) on a broadcast task.
BibTeX
@inproceedings{fainberg2019lattice,
  author = {Joachim Fainberg and Ondrej Klejch and Steve Renals and Peter Bell},
  title = {{Lattice-Based Lightly-Supervised Acoustic Model Training}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2533},
}
Figure