Developing a Stemmer for German Based on a Comparative Analysis of Publicly Available Stemmers

Leonie Weissweiler, Alexander Fraser

Abstract

Stemming is a critical component of Information Retrieval (IR) systems, yet the complex morphology of the German language presents significant challenges for traditional algorithms. This paper introduces CISTEM (Center for Information and Language Processing Stemmer), a new state-of-the-art stemmer for German. Through a comparative analysis of existing tools and the creation of two new evaluation datasets, we demonstrate that CISTEM outperforms established stemmers like Snowball in both accuracy and computational efficiency.

Introduction

In natural language processing, stemming reduces words to their base forms to improve the recall of IR systems. While English stemming is relatively straightforward, German's use of complex suffixes and prefixes makes it difficult to group related words effectively. Prior to this work, German developers lacked a systematic benchmark to compare available stemmers, many of which were inconsistent third-party implementations.

Methodology and the CISTEM Algorithm

We analyzed five prominent stemmers: Snowball, Text::German, Caumanns, and two variants of UniNE. The variations in how these tools handle German word forms are significant, as seen in the handling of word families.

Comparison of stemming results for variations of the word "Adler" (eagle) across different algorithms.

Building on the strengths of the Caumanns algorithm, we developed CISTEM with several key refinements:

  • Recursive Suffix Stripping: CISTEM iteratively removes suffixes to handle layered German endings.
  • Contextual Prefix Handling: The "ge-" prefix is stripped only if the remaining stem exceeds four characters, preventing over-stemming.
  • Case-Sensitive Rules: Specific rules, such as stripping the "t" suffix only in lowercase contexts, help distinguish between verbs and nouns.
  • Standardized Implementation: To ensure cross-platform consistency, we established a definitive sequence of operations for Perl, Python, and Java.

Evaluation Datasets

To evaluate performance, we developed two "gold standard" datasets from the CELEX2 lexical database to represent different stemming philosophies:

  1. Gold Standard 1 (Aggressive): Focuses on morphological segmentation, grouping words by stripping all non-class-bearing morphemes.
  2. Gold Standard 2 (Conservative): Based on lemma matching, grouping words that share a dictionary headword.

Results and Discussion

CISTEM demonstrated superior performance across both evaluation benchmarks, effectively bridging the "recall gap" observed in earlier tools.

Performance metrics (Precision, Recall, and F1-measure) for various stemmers against two gold standard datasets.

Performance and Recall

As shown in the results table, CISTEM achieved the highest F1-measures: 93.15% on the aggressive set and 94.40% on the conservative set. Notably, it showed a substantial improvement in recall over the widely used Snowball stemmer (89.73% vs. 83.78% recall on Gold Standard 1).

Computational Efficiency

Beyond accuracy, CISTEM is highly efficient. In runtime tests processing a corpus of approximately 624,000 words, CISTEM was roughly twice as fast as Snowball and three times faster than Text::German, making it suitable for large-scale IR applications.

Conclusion

CISTEM provides a robust, high-performance solution for German stemming. By combining recursive stripping with intelligent prefix handling and providing a bug-free reference implementation, it sets a new benchmark for the field. Future work will explore machine-learning-based rule generation and context-aware stemming for case-insensitive applications.

Summary updated

Abstract

Stemmers, which reduce words to their stems, are important components of many natural language processing systems. In this paper, we conduct a systematic evaluation of several stemmers for German using two gold standards we have created and will release to the community. We then present our own stemmer, which achieves state-of-the-art results, is easy to understand and extend, and will be made publicly available both for use by programmers and as a benchmark for further stemmer development.
BibTeX
@inproceedings{weissweiler2017developing,
  author = {Leonie Weissweiler and Alexander Fraser},
  title = {{Developing a Stemmer for German Based on a Comparative Analysis of Publicly Available Stemmers}},
  booktitle = {GSCL},
  year = {2017},
  doi = {10.1007/978-3-319-73706-5\_8},
}
Figure