A Distributed Representation-Based Framework for Cross-Lingual Transfer Parsing

Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, Ting Liu

Introduction

Dependency parsing is a fundamental task in Natural Language Processing (NLP) that identifies the grammatical structure of a sentence by determining the relationships between words. For example, in the sentence "He has good control," a parser identifies "has" as the root verb, "He" as its subject, and "control" as its direct object.

Example of a dependency parse tree showing grammatical relations and part-of-speech tags.

While high-quality parsers exist for resource-rich languages like English, most of the world's 7,000+ languages lack the large-scale, manually annotated datasets (treebanks) required for modern machine learning. This paper addresses this "low-resource" challenge by proposing a framework for cross-lingual transfer parsing.

The Challenge of Cross-Lingual Transfer

Cross-lingual transfer involves training a model on a resource-rich source language and applying it to a resource-poor target language. Historically, this required "delexicalizing" the model—stripping away actual words and relying only on universal Part-of-Speech (POS) tags (e.g., NOUN, VERB). While POS tags provide a common ground, ignoring the words themselves discards a massive amount of syntactic information. The authors bridge this "lexical gap" by mapping words from different languages into a shared, distributed vector space.

Methodology: Neural Transition-Based Parsing

The proposed framework utilizes a transition-based neural network parser. Unlike traditional linear models that rely on manually engineered discrete features, this neural architecture can automatically learn complex interactions between features.

The neural network architecture for the dependency parser, featuring lookup tables for various features and a cube activation function.

As illustrated in Figure 3, the model processes both lexical features (words, clusters) and non-lexical features (POS tags, relations, distance/valency) through specialized lookup tables. A key innovation is the use of a cube activation function ($h = (W_1 \cdot x + b_1)^3$). The authors demonstrate that this cubic non-linearity is mathematically equivalent to a low-rank tensor model, allowing the network to capture third-order feature interactions efficiently.

To ensure that word embeddings are transferable across languages, the authors propose two main induction methods:

  1. Robust Projection: Using word alignments from parallel text to project source embeddings onto target words. To handle out-of-vocabulary (OOV) words, they use a morphology-based approach that propagates embeddings based on edit distance.
  2. Canonical Correlation Analysis (CCA): Mapping pre-trained monolingual embeddings into a common space using a bilingual dictionary.

Experimental Results and Adaptation

The model was evaluated using the Universal Dependency Treebanks (UDT) v2.0 across English, German, Spanish, and French. The proposed "PROJ+Cluster" model achieved a 10.9% relative error reduction in Labeled Attachment Score (LAS) compared to standard delexicalized baselines.

Furthermore, the paper explores Target-Language Adaptation. Recognizing that languages have unique syntactic quirks (e.g., word order differences), the authors introduced a "minimal supervision" phase. By fine-tuning the transferred model on as few as 100 annotated sentences in the target language, they observed a massive jump in accuracy. For German, the Unlabeled Attachment Score (UAS) increased from 60.35% to 68.91%, nearly matching the performance of a model trained on 1,000 sentences from scratch.

Conclusion

This research demonstrates that distributed word representations are a powerful tool for cross-lingual transfer. By treating words as vectors in a shared space rather than discrete symbols, the model can leverage lexical knowledge across language boundaries, significantly improving parsing performance for low-resource languages.

Summary updated

Abstract

This paper investigates the problem of cross-lingual transfer parsing, aiming at inducing dependency parsers for low-resource languages while using only training data from a resource-rich language (e.g., English). Existing model transfer approaches typically don't include lexical features, which are not transferable across languages. In this paper, we bridge the lexical feature gap by using distributed feature representations and their composition. We provide two algorithms for inducing cross-lingual distributed representations of words, which map vocabularies from two different languages into a common vector space. Consequently, both lexical features and non-lexical features can be used in our model for cross-lingual transfer. Furthermore, our framework is flexible enough to incorporate additional useful features such as cross-lingual word clusters. Our combined contributions achieve an average relative error reduction of 10.9% in labeled attachment score as compared with the delexicalized parser, trained on English universal treebank and transferred to three other languages. It also significantly outperforms state-of-the-art delexicalized models augmented with projected cluster features on identical data. Finally, we demonstrate that our models can be further boosted with minimal supervision (e.g., 100 annotated sentences) from target languages, which is of great significance for practical usage.
BibTeX
@article{guo2016distributed,
  author = {Jiang Guo and Wanxiang Che and David Yarowsky and Haifeng Wang and Ting Liu},
  title = {{A Distributed Representation-Based Framework for Cross-Lingual Transfer Parsing}},
  journal = {J. Artif. Intell. Res.},
  year = {2016},
  doi = {10.1613/jair.4955},
}
Figure