A Hybrid Morphologically Decomposed Factored Language Models for Arabic LVCSR

Amr El-Desoky, Ralf Schlüter, Hermann NeyNAACL 2010

TL;DR

This paper tackles the challenge of modeling Arabic's complex morphology in large vocabulary continuous speech recognition (LVCSR) by combining morphological decomposition with factored language models (FLMs). Their hybrid approach achieves significant reductions in word error rate (WER) compared to traditional methods, demonstrating the power of integrating linguistic structure into language modeling.

What Problem Does This Paper Address?

Arabic, like many other languages, has a rich and complex morphological structure. This means that a single root word can be modified by numerous prefixes and suffixes to create a vast number of inflectional forms. For Automatic Speech Recognition (ASR) systems, especially those dealing with large vocabularies (LVCSR), this complexity poses a significant challenge. It leads to:

  • Poor probability estimates: The language model struggles to assign accurate probabilities to these numerous word forms, especially if they are rare or unseen in the training data.
  • High perplexity (PPL): This is a measure of how well a language model predicts a sample of text. High perplexity indicates the model is not very confident about the word sequences.
  • Increased vocabulary size: Directly modeling every possible word form leads to an unmanageably large vocabulary.

Traditional approaches to mitigate this include:

  1. Morphological Decomposition: Breaking down words into their constituent parts (e.g., prefix-stem-suffix) to model the underlying structure.
  2. Factored Language Models (FLMs): Representing words as vectors of features (factors) like stem, root, or morphological class, and modeling the probability of these factors.

This paper aims to combine the strengths of both these approaches to create more robust and accurate language models for Arabic LVCSR.

Methods

The authors propose a hybrid approach that leverages both morphological decomposition and factored language modeling. Here's a breakdown of their methodology:

  • Morphological Analysis: They use the Morphological Analyzer and Disambiguator for Arabic (MADA) to perform morphological analysis on Arabic words. MADA attaches a set of morphological tags to words in context.
  • Feature Extraction: From the MADA output, they derive four key features for each word:
  • "W" (Word): The surface form of the word.
  • "L" (Lexeme): The base form or dictionary entry of the word.
  • "M" (Morph): A description of the morphological structure (e.g., prefixes, suffixes).
  • "P" (Pattern): The word after subtracting its root (derived using linguistic knowledge).
  • Factored Language Model (FLM) Construction: Words are treated as vectors of these extracted factors. The FLM models the probability of these factors, allowing it to back off to related factors when a specific word n-gram is not observed. This is crucial for handling data sparsity.
  • FLM Optimization: They employ a Genetic Algorithm (GA-FLM) to optimize FLM parameters, including the combination of conditioning factors, backoff paths, and smoothing options. This optimization aims to minimize perplexity on a held-out dataset.
  • Hybrid Approach: The core of their contribution is using FLMs where the factors themselves are derived from morphologically decomposed words. This means the FLM operates on "factored morphemes."
  • Comparison: They compare their hybrid approach against several baselines:
  • Standard full-word n-gram models.
  • Morphologically decomposed n-gram models.
  • Standard FLMs using full-word representations.

Datasets

  • Training Data: The paper mentions using a large corpus of 206 million running full-words for FLM training.
  • Evaluation Data: They evaluate their systems on the GALE 2007 development (dev07: 2.5h) and evaluation (eval07: 4h) sets.
  • Internal Development Corpus: A small internal development corpus (40 minutes of audio) was used for selecting the best FLM topology.

Key Insights and Ideas

  • Synergy of Decomposition and Factorization: The most significant insight is that combining morphological decomposition with factored language modeling yields better results than either approach alone. This suggests that explicitly modeling the linguistic structure (decomposition) and then using that structure within a flexible probabilistic framework (FLMs) is highly effective.
  • FLMs Outperform Standard N-grams: Factored language models, even those based on full words, generally show improvements over standard n-gram models, especially when dealing with the complexities of Arabic.
  • Decomposition Benefits FLMs: While decomposition alone might not always help standard n-grams (as seen in some experiments), it significantly boosts the performance of FLMs when used to generate the factors.
  • "Factored Morphemes" are Key: The paper's success hinges on using FLMs that operate on factors derived from decomposed words, effectively treating "factored morphemes" as the units of modeling.
  • Perplexity vs. WER: The authors note that the FLM with the lowest perplexity on the held-out set did not necessarily yield the lowest word error rate. This highlights that perplexity is a useful metric for optimization but not the sole determinant of recognition performance. Higher "gtmin" values (count thresholds for backoff) can lead to better estimates and lower WER, even if they increase perplexity.

Key Results

The hybrid approach consistently demonstrated improvements in Word Error Rate (WER):

  • Compared to traditional full-word systems: They achieved a 3.5% to 7.0% relative reduction in WER.
  • Compared to non-factored decomposed systems: They achieved a 1.0% to 2.0% relative reduction in WER.
  • Specific improvements:
    • In Table 2, FLM4 (W1 M1 L1 W2 M2 L2 with gtmin=1) achieved the least WER of 19.9% on a 70k full-word lexicon, outperforming the baseline (20.4%).
  • Table 3 shows that using N-best list rescoring with a full-words based FLM improved WER by 0.8% absolute (4.8% relative) compared to standard tri-gram lattice rescoring.
  • Table 4 shows that using N-best list rescoring with a FLM based on partially decomposed words achieved an absolute improvement of 0.4% (2.7% relative) over its baseline and 2.2% (13.3% relative) compared to the original full-word baseline.
  • Table 5 indicates that even with a larger lexicon (256k), the decomposed FLM approach continued to provide improvements, achieving a 1.0% absolute (6.7% relative) reduction over the traditional full-words system.

Limitations and Future Work

  • Factored FLMs in Rescoring Phase Only: The authors acknowledge that they only considered factored language models in the rescoring phase of the ASR system. They suggest that exploring their use in earlier stages (e.g., lattice generation) could lead to further improvements.
  • Fully Decomposed Lexicons: They found that using fully decomposed lexicons (as opposed to partially decomposed) did not improve WERs, a finding they had previously demonstrated. This suggests a balance is needed.
  • Higher Order LMs: Similarly, using higher-order LMs in conjunction with fully decomposed lexicons did not yield better results.
  • Future Work: The paper implicitly suggests future work could involve integrating factored morpheme models more deeply into the ASR pipeline beyond just rescoring. Exploring different combinations of factors and optimization strategies for FLMs could also be fruitful.

Summary updated

BibTeX
@inproceedings{eldesoky2010hybrid,
  author = {Amr El-Desoky and Ralf Schlüter and Hermann Ney},
  title = {{A Hybrid Morphologically Decomposed Factored Language Models for Arabic LVCSR}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure