Introduction
Traditional language models (LMs) are typically trained on vast corpora of written text. However, this reliance on text creates a significant "text-to-speech gap." Many global languages lack a written form, and even for those that do, spontaneous speech often deviates significantly from formal writing due to disfluencies, dialects, and slang. This research addresses the challenge of learning a lexicon and an n-gram language model directly from noisy, continuous audio, bypassing the need for manual transcriptions.
Methodology: Leveraging Phoneme Lattices
The core innovation lies in the representation of speech. Rather than relying on the "one-best" output from a phoneme recognizer—which can bake in early errors—the researchers utilize phoneme lattices. A lattice is a directed acyclic graph that represents multiple possible phoneme sequences, each weighted by acoustic scores. This approach preserves the inherent uncertainty of speech recognition.
To learn from these lattices, the system employs a Hierarchical Pitman-Yor Language Model (HPYLM). This Bayesian model is particularly effective at "smoothing"—estimating probabilities for unseen words—and naturally accounts for the power-law distribution of natural language. The learning process involves:
- WFST Composition: Mathematically combining the acoustic lattice, a lexicon mapping sounds to word segmentations, and the grammar model using Weighted Finite State Transducers.
- Blocked Gibbs Sampling: An iterative process that samples possible segmentations and word sequences from the composed lattice, allowing the model to converge on a consistent lexicon and grammar.
Experimental Setup
The model was evaluated using recordings from the Japanese Diet (National Congress), a challenging real-world dataset of continuous speech. Training data sizes ranged from approximately 8 minutes to nearly 2 hours. The system's performance was compared against a "zero-gram" baseline (no linguistic info) and a "gold standard" model trained on human-generated verbatim transcriptions.
Key Results and Insights
The experiments yielded several critical findings:
- Lattices vs. One-Best: Training on lattices significantly outperformed training on the single most likely phoneme string. By preserving multiple hypotheses, the model can "recover" from acoustic errors if an alternative path provides a better linguistic fit.
- The Role of N-Grams: Higher-order models (bigrams and trigrams) were found to be essential for accurate word discovery. Simpler unigram models often suffered from "undersegmentation," incorrectly grouping common phrases into single long "words."
- Performance Scaling: As shown in the results, the proposed 3-gram model consistently improved as more data was provided, narrowing the gap with the transcript-trained model.
The learned model achieved a 7% absolute improvement in Phoneme Error Rate (PER) over the baseline with just 8 minutes of training data. While it did not reach the performance of the transcript-trained model (which was ~4% better), it demonstrated remarkable robustness to acoustic noise.
Conclusion and Future Work
This research demonstrates the feasibility of learning functional language models directly from speech. By treating word segmentation and grammar learning as a joint Bayesian inference problem over phoneme lattices, the system effectively bridges the gap between raw audio and linguistic structure. Future work will focus on reducing the computational intensity of Gibbs sampling and extending the method to fully unsupervised acoustic model training, potentially enabling the rapid development of speech technologies for under-resourced languages.
Summary updated