TL;DR
This paper introduces MorAz, an open-source morphological analyzer for Azerbaijani Turkish, addressing the need for robust linguistic tools for this agglutinative language. MorAz offers both an interactive website and a RESTful API, making its advanced morphological analysis capabilities accessible for research and integration into NLP pipelines.
What Problem Does This Paper Address?
Morphological analysis is a fundamental step in Natural Language Processing (NLP), especially for languages like Azerbaijani Turkish, which are highly agglutinative. This means words are formed by stringing together many morphemes (meaningful units), leading to a vast number of possible word forms and complex grammatical structures. Existing tools for Azerbaijani Turkish are either partial, not publicly available, or difficult to integrate. This paper aims to fill this gap by providing a comprehensive, open-source morphological analyzer that is both user-friendly and technically robust.
Methods
MorAz is built upon the well-established two-level morphology approach, utilizing Helsinki Finite-State Transducer (HFST) for its implementation. This approach separates morphotactics (word formation rules) from phonological rules. The system comprises four main FST components: nominal and verb inflections, nominal predicates, and derivation. Derivation acts as a bridge between these components.
Key methodological aspects include:
- Two-Level Morphology: A standard but powerful framework for modeling morphology, separating the abstract representation of words from their surface forms.
- HFST Implementation: Leveraging a robust and efficient finite-state transducer framework.
- Abstract Morpheme Representation: Using key-value pairs (e.g.,
<Num:Sg>) and simple keys (e.g.,<State>) to represent morphemes, allowing for optional features and clear representation of inflectional paradigms. - Manual Lexicon Curation: A significant effort was made to manually craft a minimal lexicon, aiming to reduce redundant analyses and improve accuracy by distinguishing between derived forms with clear semantic connections and those that have drifted significantly.
- Archiphonemes: The system uses archiphonemes (like 'A', 'I', 'K', 'D', 'N') to handle variations in surface forms based on phonological rules, such as vowel harmony and palatalization.
- Python and Django Wrapper: The HFST analyzer is wrapped with Python scripts and deployed on a Django web server, enabling both a user-friendly website and a RESTful API.
Datasets
The paper does not explicitly mention a training dataset. However, for evaluation, they used a test input consisting of 10,890 distinct Azerbaijani words collected from BBC Azerbaycan. This dataset was pre-processed to remove tokens starting with capital letters (to avoid issues with named entities not in the lexicon) and punctuation marks.
Key Insights and Ideas
- The Power of Open Source for Under-Resourced Languages: Providing an open-source, comprehensive morphological analyzer for Azerbaijani Turkish is crucial for advancing NLP research and applications for this language.
- Balancing Lexicon Size and Accuracy: The authors highlight the trade-off between a large, automatically generated lexicon and a manually curated one. MorAz's approach of manual curation aims to reduce spurious analyses, leading to potentially more accurate results for the covered vocabulary.
- Accessibility through Web Services: Offering both an interactive website and a RESTful API significantly lowers the barrier to entry for researchers and developers wanting to use or integrate morphological analysis into their workflows.
- Handling Agglutination: The two-level morphology approach, combined with a detailed set of rules for inflections and derivations, is well-suited to handle the complex agglutinative nature of Azerbaijani Turkish.
Key Results
- Coverage: MorAz did not return an analysis for 23.92% of the words in their test set. This indicates a gap in the lexicon or rule coverage, which is common for specialized analyzers.
- Average Analyses: For the words that were analyzed, the system provided an average of 1.96 analyses per word. This suggests a reasonable level of disambiguation or a manageable number of plausible analyses.
- Comparison with Apertium: The paper positions MorAz as more comprehensive than the partial Azerbaijani morphology implementation in Apertium, particularly in terms of coverage of morphotactics and lexicon size.
Limitations and Future Work
- Lexicon Completeness: The authors acknowledge that the lexicon is not complete, especially concerning named entities. This is an ongoing area of manual configuration and refinement.
- Future Development: The authors plan to develop an interactive tool to generate surface forms from abstract morphemes. This would be valuable for exploring the language's morphology and for educational purposes.
Summary updated