Abstract
This paper presents a novel hybrid approach to training Automatic Speech Recognition (ASR) systems by integrating "lightly-supervised" and "semi-supervised" learning. By merging inaccurate transcripts with machine-generated lattices using Finite State Transducers (FSTs), the researchers preserve uncertainty in the training data. This method achieves up to a 20% relative reduction in word error rates (WER) on broadcast data from the Scottish Parliament, demonstrating that even highly imperfect subtitles can significantly enhance model performance when processed as lattices rather than fixed text.
Introduction
High-quality ASR systems typically require vast amounts of audio paired with precise manual transcriptions. Given the prohibitive cost of manual labor, researchers often leverage "lightly-supervised" data, such as closed captions or subtitles. However, these transcripts are frequently paraphrased, incomplete, or misaligned. Conventional techniques attempt to "fix" these transcripts into a single best-guess text or filter out noisy segments. This paper argues that modern discriminative training objectives, such as Lattice-Free MMI, benefit from the probabilistic "wiggle room" provided by lattices—graphs representing multiple possible word sequences. The authors propose a method to utilize messy human transcripts while maintaining this beneficial uncertainty.
Methods
The core contribution is a Lattice Combination algorithm implemented via Finite State Transducers (FSTs). The process involves three primary components: a transcript transducer ($R$) representing the messy human transcript, a hypothesis lattice ($H$) generated by decoding the audio with a biased language model, and an edit transducer ($E$) that allows for insertions, deletions, and substitutions.
These are composed into a single structure: $T = R \circ E \circ H$. The scoring logic prioritizes matches (cost of -1) over other edits (cost of 0), forcing the algorithm to maximize the inclusion of transcript words while filling gaps with machine hypotheses.
Crucially, the resulting transducer is pruned to retain multiple likely paths, creating a "supervision lattice." This lattice "collapses" into a single path where the transcript and audio align but remains "wide" (uncertain) where they disagree. Additionally, a deletion penalty was introduced to discourage the model from skipping words, a common issue in lightly-supervised training.
Datasets and Evaluation
The models were initially trained on 179 hours of BBC news data from the Multi-Genre Broadcast (MGB) corpus. Adaptation and testing were performed on data from the Scottish Parliament, a challenging "in-the-wild" dataset. The provided subtitles for this set were highly inaccurate, with a 55.3% WER relative to a true reference. Evaluation was conducted on 6.8 hours of audio across 40 speakers.
Key Results
The proposed lattice combination method significantly outperformed standard baselines. While a standard semi-supervised approach reduced the WER from 30.0% to 28.6%, the lattice combination method achieved a WER of 23.6%. Further refinements, including a biased language model and a deletion penalty, brought the WER down to 22.8%.
These results represent a 20% relative WER reduction compared to the semi-supervised baseline, confirming that preserving uncertainty through lattices is more effective than training on "corrected" single-path text strings.
Conclusion
While effective, the current implementation of the edit transducer $E$ faces computational challenges as its search space grows quadratically with input length. Future work will focus on optimizing this through factored transducers and exploring how these improvements scale with larger datasets. The study highlights that even very messy subtitles contain valuable signals for ASR training if the system is allowed to account for their inherent uncertainty.
Summary updated