Abstract
Statistical language models face significant challenges with data sparseness, particularly in spoken language processing where high-quality training data is limited. This paper introduces the Multi-Class Composite N-gram, which combines two innovative techniques: assigning words to multiple clusters based on directional connectivity and treating frequent word sequences as single "composite" units. Experimental results demonstrate a 16% reduction in word error rate and a 40% reduction in model size compared to standard 3-gram models.
Introduction: The Challenge of Data Sparseness
N-gram models are fundamental to speech recognition but suffer from exponential parameter growth as the order increases. In spoken language contexts, many valid word sequences are absent from training sets, leading to poor predictive performance. While traditional class-based models mitigate this by grouping words, they often overlook the directional nature of word relationships—where a word's connectivity to its predecessor may differ significantly from its connectivity to its successor.
Methodology
The proposed model introduces three key advancements:
- Multi-Class Assignment: Words are assigned two distinct classes based on "positional attributes": one for preceding connectivity and one for succeeding connectivity. This captures nuances like the "a/an" distinction, where the following word's phonetic start dictates the choice.
- Composite N-grams: To capture long-distance dependencies efficiently, frequent word sequences (e.g., "in front of") are treated as single units. This allows the model to function like a higher-order N-gram for common phrases while maintaining the simplicity of a lower-order model for rare sequences.
- Automatic Clustering: The model employs an iterative algorithm to cluster words based on the Euclidean distance between transition probability vectors, ensuring clusters are optimized for the specific corpus.
Experimental Evaluation
The model was evaluated using the ATR Spoken Language Database, featuring a training set of 1.38 million words and a vocabulary of 16,531 words. Testing was conducted on a separate set of 5,880 words from 42 conversations.
Key Results
The Multi-Class Composite N-gram demonstrated superior performance across all metrics. As shown in the comparison of perplexity and parameter counts, the proposed model achieved a perplexity of 16.20, outperforming both standard 3-gram (17.88) and 4-gram (17.45) models.
Furthermore, the model significantly improved speech recognition accuracy. The composite 3-gram model achieved a word accuracy of 88.30%, representing a 16% reduction in word error rate (WER) compared to the standard word 3-gram.
Conclusion and Future Work
The Multi-Class Composite N-gram provides a highly efficient solution for spoken language modeling, offering better accuracy with fewer parameters. Future research will focus on optimizing the frequency thresholds for composite units and evaluating the model's performance on diverse linguistic structures beyond Japanese.
Summary updated