Unsupervised Cross-Lingual Adaptation of Dependency Parsers Using CRF Autoencoders

Zhao Li, Kewei TuEMNLP 2020 · OpenReview

Abstract

We consider the task of cross-lingual adaptation of dependency parsers without annotated target corpora and parallel corpora. Previous work either directly applies a discriminative source parser to the target language, ignoring unannotated target corpora, or employs an unsupervised generative parser that can leverage unannotated target data but has weaker representational power than discriminative parsers. In this paper, we propose to utilize unsupervised discriminative parsers based on the CRF autoencoder framework for this task. We train a source parser and use it to initialize and regularize a target parser that is trained on unannotated target data. We conduct experiments that transfer an English parser to 20 target languages. The results show that our method significantly outperforms previous methods.

Introduction

Dependency parsing is a critical task in natural language processing (NLP), identifying grammatical relationships between words. However, high-quality parsers typically require extensive manually annotated treebanks, which are unavailable for most of the world's languages. Cross-lingual adaptation offers a solution by transferring knowledge from a high-resource source language (e.g., English) to a low-resource target language.

Standard "Direct Transfer" (DT) methods often fail to capture target-specific linguistic nuances because they ignore unannotated data in the target language. While generative models can utilize unannotated target data, they often lack the representational power of discriminative models. This research introduces a discriminative approach using the CRF autoencoder framework to bridge this gap.

Methodology

The proposed method employs a CRF Autoencoder consisting of two primary components:

  1. Encoder: A Deep Biaffine parser that predicts dependency trees. It utilizes multilingual BERT (m-BERT) to facilitate shared representations across languages.
  2. Decoder: A component that reconstructs the original sentence's Part-of-Speech (POS) tag sequence from the predicted tree structure.

The adaptation process involves two steps. First, the model is trained in a supervised manner on the source language (English). Second, the English model initializes a target language model, which is then fine-tuned on unannotated target text by minimizing reconstruction error. To maintain stability and prevent "drift" into linguistically implausible structures, the authors implement three types of regularization:

  • Weight (W): Constraints model parameters to remain close to the source model.
  • Edge (E): Encourages similar scoring for potential dependency links.
  • Tree (T): Maintains structural similarity to the source model's expectations.
Impact of target and source corpus size on UAS.

As shown in Figure 1, even a small amount of unannotated target data (20%) significantly boosts performance, demonstrating the effectiveness of the unsupervised adaptation step.

Experimental Results

The method was evaluated on 20 languages from the Universal Dependencies (UD) v2.2 dataset, categorized into "Nearby" and "Distant" relative to English.

Comprehensive UAS comparison across 20 languages.

The results in Table 1 highlight several key findings:

  • Superiority over Generative Models: The discriminative CRF autoencoder significantly outperforms state-of-the-art generative models (e.g., an average UAS of 60.65% vs 47.73% for generative flow models on distant languages).
  • Improvement over Direct Transfer: For distant languages, the average UAS improved from 53.09% (Direct Transfer) to 60.65% (E+T regularization). For nearby languages, UAS rose from 83.09% to 86.05%.
  • Regularization Effectiveness: "Edge" (E) and "Tree" (T) regularization proved most effective, especially for distant languages where the risk of model drift is higher. Combinations like W+E or E+T often yielded the best overall performance.

Conclusion

This work demonstrates that discriminative models, when properly regularized within a CRF autoencoder framework, are highly effective for unsupervised cross-lingual dependency parsing. By leveraging unannotated target data and powerful pre-trained representations like m-BERT, the proposed method sets a new benchmark for low-resource parsing. Future work may explore more complex decoders or deeper integration with cross-lingual pre-training to further reduce reliance on gold POS tags.

Summary updated

Abstract

We consider the task of cross-lingual adaptation of dependency parsers without annotated target corpora and parallel corpora. Previous work either directly applies a discriminative source parser to the target language, ignoring unannotated target corpora, or employs an unsupervised generative parser that can leverage unannotated target data but has weaker representational power than discriminative parsers. In this paper, we propose to utilize unsupervised discriminative parsers based on the CRF autoencoder framework for this task. We train a source parser and use it to initialize and regularize a target parser that is trained on unannotated target data. We conduct experiments that transfer an English parser to 20 target languages. The results show that our method significantly outperforms previous methods.
BibTeX
@inproceedings{li2020unsupervised,
  author = {Zhao Li and Kewei Tu},
  title = {{Unsupervised Cross-Lingual Adaptation of Dependency Parsers Using CRF Autoencoders}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=pAzkxGeaIHE},
}
Figure