Abstract
This paper introduces a novel structured language model (SLM) that leverages context-sensitive, lexicalized left-corner parsing to improve word prediction in speech recognition. By combining top-down and bottom-up parsing strategies, the proposed Probabilistic Left-Corner Grammar (PLCG) model achieves greater efficiency and accuracy than traditional n-gram models or purely bottom-up parsers. Experimental results on the Penn Treebank and Wall Street Journal corpora demonstrate significant reductions in perplexity and a 4% relative reduction in Word Error Rate (WER) when the model is interpolated with standard trigram baselines.
Introduction
Standard automatic speech recognition (ASR) systems primarily rely on n-gram language models, which predict the next word based on a local window of $N-1$ preceding words. While computationally efficient, these models are inherently limited by their inability to capture long-distance dependencies and the hierarchical grammatical structures of natural language.
Structured language models (SLMs) aim to overcome these limitations by incorporating syntactic information through real-time parsing. However, previous approaches, such as the Chelba-Jelinek model, often utilized bottom-up shift-reduce parsers that can be inefficient, as they build local structures without global context. Conversely, purely top-down parsers are difficult to lexicalize effectively. This work proposes a hybrid approach using Probabilistic Left-Corner Parsing (PLCG) to create a model that is both linguistically informed and computationally viable for real-time applications.
Methodology
The core of the proposed model is a context-sensitive PLCG. This strategy initiates parsing bottom-up from an input word (the "left corner") and immediately projects potential higher-level structures top-down, effectively pruning the search space.
Key Operations and Lexicalization
The parser employs three primary operations: SHIFT (consuming the next word), PROJECT (predicting a parent constituent from a completed child), and ATTACH (connecting a completed constituent to a previously predicted goal). To enhance predictive power, the model is fully lexicalized; it "percolates" headwords up the parse tree, allowing predictions to be conditioned on "exposed heads"—the most structurally significant words from preceding constituents.
Implementation
The authors implement this model using a synchronous chart parsing algorithm. This dynamic programming approach allows the system to maintain multiple hypotheses simultaneously, ensuring that the most probable structural interpretations are tracked as the sentence unfolds word-by-word.
Experimental Setup
The model was evaluated using three key datasets:
- Penn Treebank (PTB): Used for initial training (sections 0–20) and testing (sections 23–24) on gold-standard parses.
- BLLIP WSJ Corpus (BWC): A large-scale corpus of automatically parsed Wall Street Journal articles, providing the volume of data necessary for robust language modeling.
- DARPA WSJ Nov ’92 Evaluation Set: Utilized for N-best list rescoring to measure real-world performance in a speech recognition context.
Results and Discussion
Perplexity Reductions
The PLCG-based model showed clear advantages in perplexity (PPL) over the baseline trigram. While the PLCG model alone outperformed the trigram slightly, the most significant gains were observed through linear interpolation. By combining the local predictive power of trigrams with the structural insights of the PLCG, the authors achieved a substantial drop in PPL.
Speech Recognition Performance
In rescoring tasks, the integration of the PLCG model led to a 4% relative reduction in Word Error Rate (WER). The model's higher parsing accuracy (~79% vs. ~57-75% for previous SLMs) directly translated into better features for word prediction. Furthermore, the PLCG-based SLM proved to be significantly faster than previous dynamic programming-based structured models.
Conclusion
The proposed PLCG-based structured language model successfully bridges the gap between efficient n-gram models and complex syntactic parsers. By utilizing a left-corner parsing strategy, the model captures essential structural dependencies that n-grams miss, leading to measurable improvements in both perplexity and ASR accuracy. Future work will focus on improving the robustness of the standalone model through EM re-estimation and addressing vocabulary constraints to further enhance its utility in open-domain speech recognition.
Summary updated