Language Embeddings for Typology and Cross-lingual Transfer Learning

Dian Yu, Dian Yu 0002, Taiqi He, Kenji Sagae

Introduction

Cross-lingual transfer learning aims to adapt models trained on high-resource languages, such as English, to low-resource languages where labeled data is scarce. Traditionally, this process relies on parallel corpora—direct translations that are often unavailable for the majority of the world's languages. While massively multilingual models like mBERT or XLM provide a foundation, they frequently struggle to capture the fine-grained structural differences between diverse language families.

This paper introduces a novel unsupervised method for learning "language embeddings"—dense vectors that represent a language's unique structural fingerprint—using only monolingual text. By training models on a simple word-reordering task, the researchers demonstrate that these embeddings can capture linguistic typology and significantly enhance zero-shot cross-lingual performance.

Methodology: Learning Structure through Reordering

The core of the proposed approach is a Denoising Autoencoder (DAE) trained for word reordering. The process involves four key steps:

  1. The Reordering Task: Sentences from 29 languages are randomly shuffled. The model is tasked with reconstructing the original, grammatically correct word order.
  2. Capturing Syntax: To successfully unscramble a sentence, the model must internalize the specific syntax and grammar of the target language (e.g., typical word order patterns like Subject-Verb-Object).
  3. Conditioning on Language Embeddings: The model is conditioned on a unique 50-dimensional language embedding. As the model learns to reorder sentences across multiple languages, these embeddings are forced to encode the structural characteristics necessary for the task.
  4. Shared Semantic Space: To ensure the embeddings focus on syntax rather than vocabulary, all languages are mapped into a shared English-centric word space using unsupervised MUSE embeddings. This prevents the model from identifying languages based solely on lexical differences.

Evaluation and Key Insights

The learned embeddings were evaluated on their ability to represent linguistic typology and their utility in downstream NLP tasks.

Visualizing Linguistic Families

One of the most striking results is that the unsupervised embeddings automatically organize themselves according to genetic language families. When projected into a 2D space using PCA, languages from the same family (e.g., Romance, Germanic, Slavic) form distinct clusters. This suggests that the word-reordering task effectively captures deep-seated structural similarities between related languages without any prior linguistic knowledge.

PCA projection of learned language embeddings showing automatic clustering by linguistic family.

Typology Prediction

The embeddings proved highly effective at predicting features in the World Atlas of Language Structures (WALS). They outperformed previous state-of-the-art methods and random baselines, particularly in predicting syntactic and lexical properties. Interestingly, embeddings learned through this unsupervised reordering task often surpassed those learned from parallel data (such as the Bible corpus), likely because reordering is more syntactically demanding than translation.

Downstream Task Performance

The utility of these embeddings was further validated through zero-shot cross-lingual transfer tasks.

Dependency Parsing

Integrating language embeddings into a zero-shot dependency parser led to an average improvement of 3.4 absolute points in Unlabeled Attachment Score (UAS). The gains were particularly pronounced for Romance languages; for instance, French saw an improvement of over 10 points.

Comparison of dependency parsing performance (UAS) between baseline and language embedding-enhanced models.

Natural Language Inference (XNLI)

The researchers also integrated the embeddings into large-scale transformer models like XLM and XLM-R for the Cross-lingual Natural Language Inference (XNLI) task. Even in these powerful "black box" models, the addition of explicit structural language signals consistently improved zero-shot performance across various languages.

Limitations and Future Outlook

While promising, the approach has limitations. The current dataset is heavily skewed toward Indo-European languages, and the model's performance on underrepresented groups, such as Semitic languages, was less robust. Additionally, the quality of the language embeddings is constrained by the initial word alignments provided by MUSE. Future research will focus on balancing language representation during training and improving the robustness of these embeddings for truly low-resource languages.

Summary updated

Abstract

Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.
BibTeX
@inproceedings{yu2021language,
  author = {Dian Yu and Dian Yu and Taiqi He and Kenji Sagae},
  title = {{Language Embeddings for Typology and Cross-lingual Transfer Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.560},
}
Figure