TL;DR
Combining different Hungarian corpora for training NLP models doesn't always improve performance; in fact, it can degrade it due to subtle but significant annotation differences. This paper highlights the challenges of achieving true cross-corpus compatibility, even within the same language, and calls for more detailed standardization in linguistic annotation guidelines.
What Problem Does This Paper Address?
Building robust Natural Language Processing (NLP) models, especially for tasks like morphological analysis and Part-of-Speech (POS) tagging, often relies on large amounts of annotated data. When a single corpus isn't large enough, researchers naturally consider combining multiple corpora to increase the training data size. However, the assumption that corpora annotated with the same standard (like Universal Dependencies - UD) are inherently compatible is questionable. Linguistic annotation is complex and can be subject to theoretical differences, lack of consensus among annotators, and inconsistencies introduced during data conversion. This paper investigates whether combining different Hungarian morphological corpora actually improves model performance or if annotation discrepancies create more problems than they solve.
Methods
The authors experimented with two main NLP tools: HuSpaCy (a spaCy-based pipeline for Hungarian) and PurePos (an HMM-based morphological analyzer). They employed a standard train-dev-test split methodology. Initially, they trained and tested models on each corpus individually and then in various pairwise combinations. To understand the impact of annotation differences, they conducted further experiments where one corpus was used for training and another for testing. This allowed them to analyze the specific sources of error and inconsistency, such as differences in tagsets, theoretical approaches to annotation, and issues arising from the conversion of custom annotation schemes (like emtsv) to the Universal Dependencies (UD) standard.
Datasets
The study utilized three manually annotated Hungarian corpora:
- Szeged Treebank: The largest corpus, with a significant portion automatically converted to UD.
- NYTK-NerKor: A large corpus also converted to UD.
- ELTE DH gold standard corpus: The smallest of the three, also converted to UD.
The authors note that while the UD conversion was applied to these corpora, manual checks were only performed on a small subset of the Szeged Treebank.
Key Insights and Ideas
- Compatibility is Key, Not Just Size: Simply combining corpora doesn't guarantee better performance. Inconsistent annotations can actively harm model accuracy, even if the combined dataset is larger.
- UD is Not Universally Universal: Despite the goal of Universal Dependencies, significant differences in annotation practices persist across corpora, even within the same language. This highlights the need for more detailed and unambiguous guidelines.
- Conversion Woes: Converting specialized annotation schemes (like emtsv) to UD can introduce errors and inconsistencies, especially when the original scheme has features not present in UD. This was particularly evident in how different corpora handled features like auxiliary verbs, proper nouns, and case distinctions.
- Subtle Differences Cause Big Problems: Seemingly minor differences, like the handling of emoticons or the specific criteria for auxiliary verbs, can lead to significant drops in performance when models are trained on mixed data.
- Rule-Based vs. Data-Driven: The paper touches upon the value of rule-based systems (like emMorph) for handling the vast diversity of emtsv tags, suggesting that purely data-driven approaches might struggle with such granularity.
Key Results
- Combining Corpora Degraded Performance: In most cases, training models on combined corpora led to lower accuracy compared to training on individual corpora. For example, pairing the two largest corpora (NerKor and Szeged Treebank) resulted in the worst performance.
- ELTE DH and NerKor Showed More Compatibility: These two corpora, which both used the same UD conversion tool, seemed to be more compatible with each other than other pairs.
- POS Tagging Performance: NerKor achieved the best POS tagging results when used as a training corpus.
- Lemmatization Performance: Lemmatization accuracy generally correlated with corpus size, with larger corpora yielding better results.
- Feature Performance: The F-scores for features like
NumTypeandPronTypedropped significantly when models were trained on different corpora, indicating specific annotation inconsistencies in these areas.
Limitations and Future Work
The authors acknowledge that the UD conversion process itself introduced many of the observed inconsistencies. They plan to address some of these issues in future work by:
- Extending the list of subordinate conjuncts in the UD conversion tool.
- Adding missing cases that were lost during conversion.
- Investigating and fixing obsolete features in the Szeged Treebank.
- Clarifying the annotation of participles in the ELTE DH corpus.
They also suggest that while detailed guidelines are crucial for international standards like UD, the emtsv scheme's language-specific nature makes it valuable for retaining the unique character of Hungarian corpora.
Summary updated