TL;DR
This paper introduces an unsupervised method to automatically generate morphological paradigms for new Ukrainian words, like named entities and neologisms. By leveraging unannotated corpora and basic inflection tables, it significantly expands the coverage of existing lexicons, crucial for highly inflected languages.
What Problem Does This Paper Address?
Natural language lexicons are dynamic, constantly evolving with new words and meanings. This is particularly challenging for highly inflected languages like Ukrainian, where a single word can have many forms. Existing morphological lexicons, often static and built with extensive manual effort, struggle to keep pace with these changes, leading to gaps in corpus coverage. This paper tackles the problem of efficiently extending these static lexicons to include new words, especially named entities and neologisms, without requiring costly linguistic annotation or manual updates.
Methods
The core of the proposed method is an unsupervised algorithm that takes an unannotated Ukrainian corpus and a small set of inflection tables (derived from traditional grammar textbooks) as input. The algorithm works in several stages:
- Token Splitting: It attempts to split each token in the corpus into a stem and an inflection, trying every possible inflection from the provided tables.
- Hypothesis Generation: If a split is successful, it generates a hypothetical paradigm by combining the identified stem with all other inflections from the same table.
- Hypothesis Verification: These hypothetical paradigms are then verified against the corpus by checking if a sufficient number of their predicted word forms actually appear.
- Paradigm Confirmation: The paradigm with the highest number of confirmed word forms is selected.
- Paradigm Generation: For confirmed paradigms, the algorithm generates all remaining word forms, their lemma, and associated morphological feature combinations.
A "distortion model" can optionally be incorporated to handle stem alternations, which are common in older lexical items. The approach prioritizes generating paradigms rather than just lemmas, allowing for a more robust handling of ambiguity.
Datasets
The paper utilizes several large Ukrainian corpora for evaluation:
- News Corpus: 461,451,019 words, 31,021,650 sentences.
- Wikipedia Corpus: 185,645,357 words, 15,786,948 sentences.
- Fiction Corpus: 578,988,264 words, 18,323,509 sentences.
- Law Corpus: 1,244,408,149 words, 29,208,302 sentences.
These corpora were collected as part of the lang-uk project. The baseline static morphological lexicon used for comparison is from the dict_uk project by Rysin and Starko (2019).
Key Insights and Ideas
- Lexicons are Dynamic: The paper strongly emphasizes that lexicons are not static entities but evolve rapidly, mirroring societal changes. This necessitates adaptive approaches to linguistic resource development.
- Unsupervised Learning for Inflection: The core idea is that even without explicit labels, patterns in unannotated text and basic grammatical rules can be used to infer morphological structures.
- Paradigm-Centric Approach: Focusing on generating entire paradigms, rather than just mapping words to lemmas, offers a more robust way to handle morphological variation and ambiguity.
- Complementarity of Corpora: Different types of corpora (e.g., Wikipedia vs. Law) exhibit varying degrees of lexical diversity and neologisms, making them complementary for improving morphological coverage. News and Wikipedia corpora, in particular, showed strong potential for enhancing coverage across other corpora.
- Frequency Matters: The evaluation demonstrates that the effectiveness of paradigm prediction and coverage improvement is influenced by word frequency, with higher frequencies often yielding more reliable results.
Key Results
The proposed algorithm consistently improves lexical coverage across various Ukrainian corpora.
- Wikipedia Coverage: The Wikipedia corpus, rich in named entities and specialized terminology, showed significant improvement in coverage when augmented with paradigms generated by the algorithm, especially when compared to the baseline static lexicon. For instance, in Figure 4, combining the Wiki lexicon with generated paradigms from the News corpus significantly reduced the percentage of non-covered types.
- News and Wikipedia Synergy: The News and Wikipedia corpora were found to be highly complementary, with each improving the coverage of the other.
- Improvement Rates: Figures 10 and 11 show improvement rates, with the highest improvements observed for the News and Wiki corpora, reaching up to 16.3% for lexical items with frequencies of 10 and higher when using the Wiki corpus to test the News corpus.
- Law and Fiction Corpora: The Law and Fiction corpora were harder to improve, likely due to their lower lexical diversity and fewer neologisms.
Limitations and Future Work
- Irregular Paradigms: The current algorithm primarily focuses on regular dynamic components of the lexicon and does not predict irregular paradigms.
- Accuracy Evaluation: While coverage is evaluated, direct accuracy evaluation of generated paradigms is noted as a limitation, requiring more complex metrics and manual annotation.
- Distortion Models: The paper suggests future work on developing distortion models to cover less regular cases and stem alternations.
- Contextual and Syntactic Features: The authors propose exploring the use of contextual and syntactic features to further verify predicted morphological properties.
- Frequency Thresholds: Further research is needed to experimentally establish optimal frequency and coverage thresholds for different error rates, especially for less frequent items.
Summary updated