TL;DR
This paper presents a system for contextual morphological analysis and lemmatization that achieved top results in the SIGMORPHON 2019 Shared Task. By integrating pre-trained contextualized embeddings (BERT), using individual morphological features for regularization, and merging corpora for certain languages, their system significantly improved lemmatization accuracy and achieved a close second in morphological analysis.
What Problem Does This Paper Address?
Accurately analyzing and lemmatizing text, especially for morphologically rich languages, remains a significant challenge in Natural Language Processing (NLP). Morphologically rich languages have complex word structures where a single word can convey a lot of grammatical information (e.g., tense, number, gender). The SIGMORPHON 2019 Shared Task specifically focused on "Contextualized Morphological Analysis and Lemmatization," aiming to improve these tasks by leveraging context and richer linguistic features. Existing systems often struggle with the vast number of possible word forms and their corresponding analyses, especially when dealing with diverse languages and limited training data for some. This paper addresses the need for more robust and accurate morphological analysis and lemmatization systems that can handle this complexity.
Methods
The authors built upon their previous UDPipe 2.0 system, a strong performer in previous NLP tasks. They introduced three key improvements:
- Contextualized Embeddings (BERT): They incorporated pre-trained BERT embeddings as an additional input to their neural network. This allows the model to capture word meanings based on their surrounding context, which is crucial for disambiguating word forms and their analyses. They used different BERT models (native English/Chinese, multilingual) depending on the language.
- Regularization with Individual Morphological Features: Instead of just predicting the entire set of morphological features (the POS tag) at once, they also trained the model to predict individual morphological features. This acts as a form of regularization, guiding the model to learn more accurate feature predictions and improving overall performance, especially for languages with complex morphology.
- Corpora Merging: For languages with multiple available corpora, they experimented with merging these corpora to create a larger, unified training dataset. This approach aims to leverage more data, though they found it sometimes led to performance degradation due to annotation differences, which they addressed during inference.
They also employed model ensembling, combining predictions from multiple trained models to achieve the best results.
Datasets
The research utilized data from the SIGMORPHON 2019 Shared Task, which comprised 66 distinct languages across 107 corpora. The task involved processing text in CONLL-U format. For pre-trained embeddings, they referenced FastText embeddings trained on CoNLL 2017 UD Shared Task data and also used embeddings from Grave et al. (2018). BERT embeddings were used as provided by Google.
Key Insights and Ideas
- Context is King (for Morphology): The integration of BERT embeddings significantly boosted performance, highlighting the importance of contextual information for disambiguating morphological features and lemmas.
- Fine-grained Supervision Helps: Predicting individual morphological features, even if not used during inference, acted as a powerful regularizer, leading to substantial gains in both lemmatization and POS tagging accuracy. This suggests that breaking down complex prediction tasks into smaller, related sub-tasks can improve overall learning.
- Data Merging is Tricky: While merging corpora for the same language seems intuitive, it's not a silver bullet. Differences in annotation schemes across corpora can negatively impact performance, requiring careful handling during inference.
- Ensembling for Robustness: Combining multiple models, even those with slight variations, proved to be a highly effective strategy for achieving top performance, demonstrating the benefits of diverse model perspectives.
Key Results
The system achieved outstanding results in the SIGMORPHON 2019 Shared Task:
- Lemmatization: The system achieved a lemmatization accuracy of 95.78%, outperforming all other submitted systems by a significant margin. The second-best system scored 95.00%.
- Morphological Analysis: The system placed a very close second in morphological analysis accuracy with 93.19%, just shy of the winning system's 93.23%.
Ablation experiments (Table 3) showed that each of the proposed improvements (BERT embeddings, feature regularization) contributed positively to the performance, with their combination yielding the highest scores.
Limitations and Future Work
The authors acknowledge that while their competition entry achieved excellent results, a post-competition analysis suggested that their specific ensembling strategy might have overfit to POS tag prediction due to noise. They propose a more robust ensembling method for future work. They also note that while corpora merging can be beneficial, it requires careful handling of annotation inconsistencies. Future work could explore more sophisticated methods for dealing with cross-corpus variations and further investigate the impact of contextualized embeddings across a wider range of languages and morphological complexities.
Summary updated