TL;DR
This paper presents a novel hybrid approach to automatically enhance a German morphological database by combining existing tools (SMOR, Moremorph) with a large Wikipedia corpus. The method significantly improves the coverage and precision of morphological analysis for German words, particularly complex compounds, leading to a more robust resource for downstream NLP tasks.
What Problem Does This Paper Address?
German, unlike languages like English, doesn't use spaces to delineate word boundaries, leading to long, complex words formed through compounding and derivation. This makes morphological analysis and segmentation a significant challenge. Existing tools often produce numerous ambiguous analyses, and while some databases exist for flat word segmentation, a resource for deep, linguistically meaningful parsing of German words has been lacking. This paper aims to fill that gap by creating a more comprehensive and accurate morphological database.
Methods
The authors propose a hybrid approach that leverages a multi-stage process:
- Database Look-up: The primary method involves checking if a word or its sub-parts exist in their established morphological tree database or a list of monomorphemes.
- SMOR/Moremorph Integration: If the database lookup fails, they utilize the SMOR morphological analyzer and its add-on, Moremorph, to generate initial analyses.
- Contextual Evaluation: For words not fully covered by the above, a data-intensive approach is employed. This involves:
- Building Combinations: Generating all plausible combinations of sub-analyses from Moremorph.
- Contextual Search: Using a large Wikipedia corpus, they analyze the co-occurrence of proposed word components within similar contexts. The hypothesis is that correct segmentations will have components appearing in closer proximity and more frequently together than incorrect ones.
- Document Frequency Scoring: A scoring mechanism based on document frequencies (df) and text frequencies (Stf) is used to rank hypothetical analyses.
- Filtering Implausibles: Linguistically implausible splits (e.g., a hypothetical subpart ending with a prefix) are filtered out.
- Incremental Enrichment: New analyses are added to a "new splits" database and can be used to update the main database, leading to a continuously improving resource.
Datasets
- German Morphological Tree Database: This database is built upon the refurbished CELEX database for German morphology and compound analyses from the GermaNet database.
- Wikipedia Corpus: A large, freely available, and annotated German Wikipedia corpus (2015 edition, 1.8 million articles) was used for contextual evaluation. It was tokenized and lemmatized.
- Korpus Magazin Lufthansa Bordbuch (MLD): This in-flight magazine corpus was used for testing and evaluation. It comprises 276 texts with 5,202 paragraphs, 16,046 sentences, and 260,114 tokens.
Key Insights and Ideas
- Hybrid Approach is Key: Combining rule-based tools (SMOR/Moremorph) with data-driven corpus analysis (Wikipedia) is crucial for achieving high coverage and accuracy in German morphological parsing.
- Contextual Co-occurrence as a Signal: The frequency and proximity of proposed word components within a large corpus serve as a powerful signal for disambiguating complex German words.
- Incremental Database Building: The proposed method allows for the continuous enrichment of morphological databases by incorporating newly discovered analyses, leading to a self-improving system.
- Addressing Systematic Errors: The paper identifies systematic errors (e.g., with verb particles) and proposes future work to address them, highlighting the iterative nature of NLP research.
Key Results
- Coverage: The combined approach achieved a coverage of 99% of lemma types in the MLD corpus. The initial database look-up covered 55.99% of lemma types, with SMOR/Moremorph handling an additional portion, and the data-intensive methods covering the rest.
- New Instances: Over 5,000 new instances of complex words were derived.
- Precision: The precision of new complex splits and monomorphemes was reported to be between 0.93 and 0.99. Specifically, for the test corpus, precision was 0.935 for new splits and 0.998 for newly found monomorphs.
- SMOR/Moremorph Quality: The authors found only 26 wrongly segmented words in a sample of a thousand words from the SMOR/Moremorph output, indicating good baseline quality.
Limitations and Future Work
- Systematic Errors: The paper acknowledges systematic errors, particularly with verb particles that are homographs of prepositions.
- Future Work:
- Addressing Verb Particle Errors: Future research will focus on adjusting the analysis by incorporating factors related to word length and frequency (e.g., Zipf's law) to penalize short, frequent morphemes and give more weight to longer, rarer components.
- Augmenting Lexical Databases: The long-term direction is to create more complex analyses from existing ones and further augment lexical databases.
- Creating Complex Analyses: The goal is to build complex analyses out of existing ones and augment lexical databases.
Summary updated