A Freely Available Morphological Analyzer, Disambiguator and Context Sensitive Lemmatizer for German

Wolfgang Lezius, Reinhard Rapp, Manfred WettlerCOLING 1998

TL;DR

This paper introduces Morphy, a freely available, integrated software package for German morphology, part-of-speech (POS) tagging, and context-sensitive lemmatization. It addresses the lack of comprehensive, open-source morphological tools for German, offering a large lexicon and robust algorithms that significantly improve upon existing solutions.

What Problem Does This Paper Address?

Morphological analysis is a foundational task in Natural Language Processing (NLP), crucial for applications like parsing, machine translation, and indexing. However, many existing morphological systems are proprietary, making them inaccessible for research and development. While some open-source efforts exist, their lexicons are often small, limiting their coverage. For German, a highly inflectional language, this gap is particularly pronounced. The authors aim to fill this void by providing a comprehensive, freely available tool that handles German morphology, disambiguates word senses through POS tagging, and accurately lemmatizes words in context.

Methods

Morphy is presented as a three-part integrated system:

  • Lexical System: This module stores base forms and inflection classes. Its key innovation is a compact representation that allows for a large lexicon (324,000 word forms from 50,500 stems) to fit within a small disk space. Crucially, it's designed for user-friendly extension, allowing individuals without deep linguistic expertise to add new words.
  • Generation Module: This component produces all inflected forms from a given root and inflection type, aiming for 100% accuracy. It handles complex German morphological phenomena like vowel mutation, shifts between 'ß' and 'ss', and various prefix/suffixations for participles.
  • Analysis Module: This module determines the root, POS, and grammatical features (gender, case, number, etc.) for each word form. It also handles German compound nouns using a longest-matching rule. A key aspect is its ability to segment compound nouns, even those with ambiguous linking letters or irregular compounding.
  • Part-of-Speech (POS) Tagger: To resolve ambiguities arising from the morphological analysis (where a single word form can have multiple readings), Morphy incorporates a statistical POS tagger. The authors chose a trigram algorithm, similar to Church (1988), for its simplicity, speed, and robustness.
  • Context-Sensitive Lemmatizer: This final component uses the output of the POS tagger to select the correct lemma for ambiguous word forms, ensuring that the inflected form agrees with the determined grammatical category.

Datasets

  • Lexicon Development: The lexicon was checked for completeness against the Wahrig Deutsches Wörterbuch (1997), which contains 120,000 headwords.
  • Training Corpus: A portion of the Frankfurter Rundschau Corpus (approximately 20,000 words) was used for training the POS tagger. This corpus was manually tagged by selecting the correct choice from the possibilities generated by the morphological analyzer.
  • Evaluation Datasets:
    • A test corpus of about 7,800 word forms, comprising two political speeches, a fragment of the LIMAS-corpus, and a list of special word forms, was used to evaluate the morphological analyzer's coverage. This corpus was also used in the "Morpholympics" competition.
  • A 5000-word test sample from the Frankfurter Rundschau-Corpus (distinct from the training data) was used to evaluate the POS tagger.
  • Another test sample from the Frankfurter Rundschau-Corpus was used to evaluate the lemmatizer's performance.

Key Insights and Ideas

  • Compact Lexicon Design: The paper highlights the effectiveness of storing only base forms and inflection classes to create a highly compact yet comprehensive lexicon, making it suitable for PC-based and internet-distributed systems.
  • User-Extensible Lexicon: The emphasis on a user-friendly lexicon extension mechanism is a significant practical advantage, democratizing the ability to improve morphological coverage.
  • Integrated Approach: The synergy between the morphological analyzer, POS tagger, and lemmatizer is key. The tagger's context-aware disambiguation is essential for the lemmatizer to select the correct root from multiple possibilities.
  • Trade-offs in Tagging: The evaluation of different tag sets (large vs. small) for lemmatization reveals an interesting trade-off. While a larger tag set offers finer distinctions, a smaller, more general set can lead to higher tagging accuracy due to reduced data sparsity, which in turn benefits lemmatization.

Key Results

  • Morphological Coverage: The current version of Morphy achieved high recognition rates on the test corpus: 94.3%, 98.4%, 96.2%, and 88.9% for different parts of the corpus. This represents a significant improvement over the older version, especially considering the larger lexicon.
  • POS Tagging Accuracy: The trigram tagger achieved 84.7% accuracy on a 5000-word test sample using a large tag set. When mapped to a smaller set of 51 tags, accuracy improved to 95.9%.
  • Lemmatization Accuracy: The lemmatizer, using the POS tagger's output, achieved an overall accuracy of about 99.3%. The small tag set yielded slightly better lemmatization results (89.9% of ambiguous word forms correctly lemmatized) compared to the large tag set (88.6%).

Limitations and Future Work

  • Limitations:
    • The morphological algorithms are complex and not fully described in detail.
  • The implementation of derivation rules is only partially completed.
  • Compound noun processing, while handled, might not include all cases where compounds are not typically broken down.
  • The current POS tagger's accuracy, while good, could still be improved.
  • The paper notes that semantic ambiguities cannot always be resolved by syntactic considerations alone.
  • Future Work:
    • The authors suggest that future improvements in tagging accuracy, particularly with the large tag set, might eventually make it superior for lemmatization.
  • They are also developing a rule-based tagging system, which they believe could further reduce error rates.
  • The possibility of including mood in the tag set is mentioned as a way to resolve certain ambiguities.
  • The system is intended for Windows NT/95, with an export facility for UNIX users to generate full-form lexicons. Future work might involve broader platform support.

Summary updated

BibTeX
@inproceedings{lezius1998freely,
  author = {Wolfgang Lezius and Reinhard Rapp and Manfred Wettler},
  title = {{A Freely Available Morphological Analyzer, Disambiguator and Context Sensitive Lemmatizer for German}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1998},
}
Figure