TL;DR
This paper tackles the challenge of dependency parsing for Croatian and Serbian, two under-resourced, morphologically rich South Slavic languages. By leveraging existing Croatian dependency treebanks and applying state-of-the-art parsing techniques, the authors achieve competitive parsing accuracy for both languages, demonstrating the potential for resource sharing and transfer.
What Problem Does This Paper Address?
Natural Language Processing (NLP) for morphologically complex languages with relatively free word order, like Croatian and Serbian, presents significant challenges. These languages are considered "under-resourced," meaning they lack the extensive annotated data and pre-trained models common for languages like English. Specifically, dependency parsing – a crucial task for understanding sentence structure – has been largely unaddressed for Serbian, and existing work for Croatian is limited. The authors aim to bridge this gap by developing robust dependency parsing models for both languages, investigating the impact of preprocessing and feature selection, and exploring the feasibility of using resources developed for one language to parse the other.
Methods
The core of the methodology involves using the MSTParser, a non-projective maximum spanning tree parser known for its effectiveness with languages like Croatian. The authors experiment with three different dependency treebank formalisms: HOBS, HOBS + Sub, and SETIMES.HR. They investigate the impact of:
- Formalism Selection: Comparing parsing performance across the three different treebank structures.
- Preprocessing: Evaluating the effect of automatic lemmatization and morphosyntactic (MSD) tagging on parsing accuracy, using publicly available models.
- Morphosyntactic Feature Selection: Systematically removing individual features from the MSD tagset to understand their contribution to parsing performance.
Datasets
The study utilizes two Croatian dependency treebanks:
- HOBS (Croatian Dependency Treebank): Available in two instances (basic and with subordinate clause tags).
- SETIMES.HR Treebank: A smaller, newer treebank.
The paper also uses publicly available test sets for Croatian and Serbian, originally created for lemmatization and tagging experiments. These test sets are derived from two domains:
- Newspaper text (set.test)
- Wikipedia (wiki.test)
The SETIMES.HR treebank was expanded by 50% for this experiment.
Key Insights and Ideas
- Resource Sharing is Viable: The study strongly suggests that parsing models trained on Croatian data can be reliably used for Serbian, indicating a low need for syntactic annotation projection between these closely related languages.
- SETIMES.HR Outperforms: The SETIMES.HR formalism consistently achieved the highest parsing accuracy, outperforming the HOBS-based models by a significant margin. This is attributed to its simpler, yet informative, syntactic tagset and its formalism being better suited for Croatian syntax.
- Preprocessing Matters, But Not Critically: While manual (gold standard) preprocessing yields better results, automatic preprocessing (lemmatization and MSD tagging) only reduces LAS scores by 3-4 points and UAS by 2 points. This is considered favorable for complex NLP systems.
- Morphosyntactic Features are Crucial: The impact of morphosyntactic tagging is substantial. Removing specific features, particularly noun case, significantly degrades parsing accuracy, highlighting the importance of rich morphological information. Interestingly, removing adjective case improved parsing accuracy, suggesting potential for feature optimization.
- Language Differences are Subtle in Parsing: Cross-language differences in parsing accuracy were minimal, suggesting that the models are robust to the slight variations between Croatian and Serbian.
Key Results
- Top Performance: The SETIMES.HR model achieved the highest overall LAS score of 74.3% on Croatian and 74.1% on Serbian test sets.
- Formalism Advantage: SETIMES.HR outperformed HOBS + Sub by approximately 9 LAS points and HOBS by 17 LAS points.
- Preprocessing Impact: Automatic preprocessing reduced LAS by 3-4 points and UAS by 2 points compared to manual preprocessing.
- Feature Impact: Removing noun case led to the most substantial decrease in parsing accuracy. Removing adjective case resulted in an accuracy increase.
Limitations and Future Work
- Test Set Size: The authors acknowledge that the test set size (100 sentences per sample) might be small compared to standard CoNLL shared tasks, but they mitigated this by merging test sets.
- Domain Adaptation: The observed accuracy decrease when moving from newspaper text to Wikipedia suggests a need for domain adaptation strategies for parsing.
- Advanced Parsers: The authors plan to experiment with newer, more advanced dependency parsers to obtain up-to-date scores.
- Morphosyntactic Tagset Design: Future work will focus on optimizing morphosyntactic tagset design for improved dependency parsing, investigating feature influences and dependencies.
- Extrinsic Evaluation: The paper suggests that extrinsic evaluation would be beneficial to further assess the utility of the developed models.
Summary updated