Morpheme level feature-based language models for German LVCSR

Amr El-Desoky Mousa, M. Ali Basha Shaik, Ralf Schlüter, Hermann NeyInterspeech 2012

TL;DR

This paper tackles the challenge of German's complex morphology in Large Vocabulary Continuous Speech Recognition (LVCSR) by proposing morpheme-level feature-based language models. By combining the benefits of morpheme decomposition and feature-rich modeling, they achieve significant reductions in Word Error Rate (WER) compared to previous state-of-the-art methods.

What Problem Does This Paper Address?

German, with its rich morphology and tendency for compounding words, presents a significant hurdle for LVCSR systems. This complexity leads to a high number of unique word forms, resulting in data sparsity, increased Out-of-Vocabulary (OOV) rates, and consequently, poor language model probabilities and higher perplexities. Traditional approaches like using larger vocabularies or more training data can be resource-intensive and don't fully resolve the OOV issue. This paper aims to improve LVCSR performance for German by developing language models that better handle its morphological richness.

Methods

The core of this work lies in combining two key ideas: morpheme-level modeling and feature-based language models.

  • Morpheme Decomposition: Words are first broken down into their constituent morphemes using Morfessor, a data-driven tool that discovers optimal segmentations based on the Minimum Description Length (MDL) principle. This process helps to reduce data sparsity and improve lexical coverage.
  • Feature Derivation: For both full words and derived morphemes, features are extracted. These include:
  • Part-of-Speech (POS): Grammatical category of the word/morpheme.
  • Lemma (Baseform): The canonical form of the word.
  • Index: A data-driven class index derived from word-pair co-occurrence matrices and Singular Value Decomposition (SVD), clustered using k-means.
  • Language Model Types: The paper compares three types of feature-based language models:
  • Stream-based Language Models (SLMs): N-gram models built over sequences of selected feature streams (e.g., sequences of baseforms or POS tags).
  • Class-based Language Models (CLMs): Models that combine N-gram models over classes with word probabilities within those classes, allowing for multiple class memberships.
  • Factored Language Models (FLMs): These models use a complex backoff mechanism across multiple features (word, baseform, POS) to estimate probabilities, allowing for interdependencies between features.
  • Model Combination: The authors also explore interpolating N-gram LMs with CLMs and combining scores from different LMs for N-best list rescoring.

Datasets

  • LM Training Corpus: Approximately 188 million running full-words, including data from the Quaero project, Broadcast News (BN), European Parliament Plenary Sessions (EPPS), read articles, dialogs, and web data.
  • Evaluation Corpora: Quaero 2009 development (dev09: 7.5h) and evaluation (eval09: 3.8h) corpora, consisting of audio material from EPPS sessions, web sources, and some BN data.

Key Insights and Ideas

  • Morphemes as a Solution to German Morphology: Decomposing words into morphemes is a viable strategy to combat the data sparsity and OOV issues inherent in German LVCSR.
  • Feature-Rich Morpheme Models: Applying feature-based modeling techniques, typically used for full words, to morphemes themselves yields significant benefits. This allows for better generalization and smoothing.
  • FLMs vs. CLMs vs. SLMs: While all feature-based models show improvement, the paper highlights that CLMs and FLMs generally outperform SLMs. FLMs, with their complex backoff, are particularly effective.
  • Interpolation is Key: Interpolating class-based LMs with traditional N-gram models (both word and morpheme-based) proves to be a highly effective strategy for improving recognition performance.
  • Feature Selection Matters: The "Index" feature, while useful for CLMs, was found not to be helpful for the FLM in this study.

Key Results

The paper reports significant reductions in Word Error Rate (WER):

  • Compared to traditional 4-gram lattice rescoring: Relative WER reductions of 0.9% (dev09) and 1.8% (eval09) for full-word systems, and 1.5% (dev09) and 1.8% (eval09) for morpheme-based systems.
  • Compared to previous best results using FLMs: Relative WER reductions of around 1.5% are achieved.
  • Overall Best Performance: Achieved by interpolating all class-based models with the traditional word/morpheme N-gram models, leading to relative WER reductions of around 3.0% compared to the conventional word-based approach.
  • Statistical Significance: The reported WER reductions are statistically significant with a probability of improvement around 95%.

Limitations and Future Work

  • Score Combination Limitations: The paper notes that attempting to perform score combination of FLMs with interpolated CLMs did not further reduce WERs in their experiments.
  • Future Work: While not explicitly detailed, the success of morpheme-level feature-based models suggests further exploration into different feature combinations, decomposition methods, and advanced model interpolation or combination techniques. The authors' previous work also hints at the potential of sub-lexical language models.

Summary updated

BibTeX
@inproceedings{mousa2012morpheme,
  author = {Amr El-Desoky Mousa and M. Ali Basha Shaik and Ralf Schlüter and Hermann Ney},
  title = {{Morpheme level feature-based language models for German LVCSR}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure