Language Embeddings for Typology and Cross-lingual Transfer Learning

Dian Yu, Taiqi He, Kenji Sagae

Abstract

Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.

Introduction

In Natural Language Processing (NLP), cross-lingual transfer aims to apply models trained on high-resource languages to low-resource ones. Traditional methods rely heavily on parallel corpora or hand-curated linguistic databases like the World Atlas of Language Structures (WALS). However, most of the world's 7,000+ languages lack such resources. This research investigates whether a language's structural "DNA"—its syntax and word-order rules—can be learned from unannotated text alone.

Methodology: Learning Syntax through Shuffling

The core innovation is a Denoising Autoencoder (DAE) trained on a reordering task. Instead of standard language modeling, the model is presented with sentences where word order has been randomly shuffled. To reconstruct the original sentence, the model must understand the specific grammatical rules of the target language.

A 50-dimensional language embedding is appended to every input word. To ensure the model captures structural rather than just lexical information, MUSE (unsupervised word embeddings) is used to map all 29 languages into a shared semantic space. This forces the language embedding to encode the syntactic patterns required to "unshuffle" the text.

Typological Analysis

The learned embeddings were evaluated for their ability to capture real-world linguistic relationships. When visualized using Principal Component Analysis (PCA), the embeddings naturally cluster into recognized language families such as Romance, Germanic, and Slavic, without any explicit supervision regarding language genealogy.

PCA visualization of 50-dimensional language embeddings, showing natural clustering by linguistic families.

Furthermore, these unsupervised vectors proved highly effective at predicting WALS features, particularly those related to syntax, outperforming previous methods that required parallel data.

Cross-lingual Performance

The utility of these embeddings was tested in zero-shot settings for two major tasks: dependency parsing and Natural Language Inference (NLI).

Dependency Parsing

In cross-lingual dependency parsing (Universal Dependencies v2.6), adding language embeddings resulted in an average improvement of 3.4 points in Unlabeled Attachment Score (UAS). The gains were particularly pronounced in Romance languages, with French seeing an improvement of over 10 points.

Zero-shot dependency parsing performance (UAS) comparing the baseline to models augmented with learned language embeddings.

Natural Language Inference

For the XNLI task, integrating these embeddings into state-of-the-art multilingual models like XLM and XLM-R provided consistent performance boosts (up to 1.1 absolute points on average), demonstrating that structural language representations can enhance high-level semantic reasoning across languages.

Conclusion and Limitations

This work demonstrates that meaningful language representations can be learned without supervision. While highly effective for many language families, the model showed limitations with Semitic languages like Hebrew, likely due to challenges in unsupervised word alignment for those pairs. Future research will focus on expanding the diversity of the training set and integrating these embeddings into larger Transformer architectures to further bridge the gap for low-resource languages.

Summary updated

Abstract

Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.
BibTeX
@inproceedings{yu2021language,
  author = {Dian Yu and Taiqi He and Kenji Sagae},
  title = {{Language Embeddings for Typology and Cross-lingual Transfer Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.560},
}
Figure