Cross-Lingual Lemmatization and Morphology Tagging with Two-Stage Multilingual BERT Fine-Tuning

Dan KondratyukACL 2019

TL;DR

This paper tackles the challenge of cross-lingual lemmatization and morphology tagging by fine-tuning a multilingual BERT model. They introduce a novel two-stage training approach: first, a broad multilingual fine-tuning across all available treebanks, followed by individual treebank fine-tuning. This method significantly boosts performance, achieving top results in the SIGMORPHON 2019 Shared Task.

What Problem Does This Paper Address?

Morphological analysis and lemmatization are fundamental tasks in natural language processing, crucial for understanding word forms and their dictionary entries. However, performing these tasks accurately across many different languages, especially those with limited data, is a significant challenge. Traditional approaches often require language-specific models, which are costly to develop and maintain. The SIGMORPHON 2019 Shared Task specifically aimed to address this by requiring systems to handle 107 treebanks across 66 languages, using the UniMorph schema for morphological descriptions. The core problem is how to leverage the power of large pre-trained language models like BERT to achieve high performance on these tasks in a cross-lingual setting, overcoming data scarcity and language diversity.

Methods

The authors build upon the UDify framework, which uses a shared BERT encoder for multiple NLP tasks. Their approach involves a two-stage fine-tuning strategy for a multilingual BERT base cased model:

  1. Stage 1: Multilingual Fine-tuning (MULTI): The BERT model is fine-tuned on a concatenated dataset of all available treebank training data. This stage aims to learn cross-lingual representations and generalize across languages.
  2. Stage 2: Monolingual Fine-tuning (MONO): After the initial multilingual fine-tuning, the model is further fine-tuned on each treebank individually. Crucially, this stage uses the weights from the multilingual fine-tuning as a starting point. This is done to adapt the model to the specific nuances of each language and treebank.

The model architecture includes:

  • A shared BERT encoder to process input sentences.
  • Task-specific layer attention (similar to ELMo) to generate embeddings for lemmatization and morphology tagging separately.
  • Character-level embeddings using a bidirectional LSTM to capture sub-word information.
  • Word-level bidirectional LSTMs for both lemmatization and morphology tagging decoders.
  • Feedforward layers for predicting edit scripts (for lemmatization) and morphology tags.

They also employ discriminative fine-tuning where different groups of BERT layers have different learning rates, and regularization techniques adapted from UDify.

Datasets

The primary datasets used are the treebanks provided by the Universal Dependencies (UD) project. These treebanks were converted from UD format to the UniMorph schema for morphological analysis and lemmatization. The system was trained and evaluated on 107 separate treebanks, representing 66 different languages. No new benchmarks were introduced; the focus was on leveraging existing UD treebanks for the SIGMORPHON 2019 Shared Task.

Key Insights and Ideas

  • Two-stage fine-tuning is superior: The most significant insight is that a two-stage approach (multilingual pre-training followed by monolingual fine-tuning) consistently outperforms a single-stage monolingual fine-tuning (MONO) or a purely multilingual fine-tuning (MULTI). This suggests that leveraging cross-lingual knowledge first, then specializing, is highly effective.
  • Cross-lingual information boosts performance: Even for languages with only one treebank, the MULTI and MULTI+MONO models show improvements, indicating that the multilingual pretraining itself provides valuable regularization and cross-lingual transfer.
  • Character-level and word-level LSTMs are beneficial: The addition of character-level and word-level LSTMs, when combined with the two-stage fine-tuning, further enhances performance beyond what BERT alone can achieve.
  • MULTI alone is not optimal: Interestingly, the MULTI configuration (fine-tuning only on the concatenated multilingual data without a second monolingual stage) performed worse than the MONO configuration on many treebanks. This highlights the necessity of the second, specialized fine-tuning stage.

Key Results

The proposed MULTI+MONO system achieved outstanding results in the SIGMORPHON 2019 Shared Task:

  • Highest average accuracy and F1 score in morphology tagging among all submissions.
  • Second highest average lemmatization accuracy.
  • The paper reports average scores across configurations, with MULTI+MONO achieving 95.00 accuracy and 93.23 F1 for morphology tagging, significantly outperforming the baseline (73.16 accuracy, 87.92 F1).
  • Detailed results in the tables show that MULTI+MONO consistently outperforms MONO across a vast majority of the 107 treebanks.

Limitations and Future Work

  • No extensive hyperparameter search or ensembling: The authors acknowledge that their results could be "raised much higher" by performing more extensive hyperparameter tuning and employing ensembling techniques.
  • Understanding the contribution of each component: While they show the effectiveness of the two-stage training and additional layers, a more detailed analysis of the exact contribution of multilingual fine-tuning versus BERT pretraining to the overall performance is left for future work.
  • Investigating the bottleneck of MULTI: The paper notes that the MULTI configuration alone performed poorly and suggests that the LSTMs and character embeddings might act as an information bottleneck in this scenario, requiring reconfiguration for better multilingual data integration. This warrants further investigation.

Summary updated

BibTeX
@inproceedings{kondratyuk2019cross,
  author = {Dan Kondratyuk},
  title = {{Cross-Lingual Lemmatization and Morphology Tagging with Two-Stage Multilingual BERT Fine-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure