Abstract
Cross-lingual language tasks typically require a substantial amount of annotated data or parallel translation data. We explore whether language representations that capture relationships among languages can be learned and subsequently leveraged in cross-lingual tasks without the use of parallel data. We generate dense embeddings for 29 languages using a denoising autoencoder, and evaluate the embeddings using the World Atlas of Language Structures (WALS) and two extrinsic tasks in a zero-shot setting: cross-lingual dependency parsing and cross-lingual natural language inference.
Introduction
In Natural Language Processing (NLP), cross-lingual transfer aims to apply models trained on high-resource languages to low-resource ones. Traditional methods rely heavily on parallel corpora or hand-curated linguistic databases like the World Atlas of Language Structures (WALS). However, most of the world's 7,000+ languages lack such resources. This research investigates whether a language's structural "DNA"—its syntax and word-order rules—can be learned from unannotated text alone.
Methodology: Learning Syntax through Shuffling
The core innovation is a Denoising Autoencoder (DAE) trained on a reordering task. Instead of standard language modeling, the model is presented with sentences where word order has been randomly shuffled. To reconstruct the original sentence, the model must understand the specific grammatical rules of the target language.
A 50-dimensional language embedding is appended to every input word. To ensure the model captures structural rather than just lexical information, MUSE (unsupervised word embeddings) is used to map all 29 languages into a shared semantic space. This forces the language embedding to encode the syntactic patterns required to "unshuffle" the text.
Typological Analysis
The learned embeddings were evaluated for their ability to capture real-world linguistic relationships. When visualized using Principal Component Analysis (PCA), the embeddings naturally cluster into recognized language families such as Romance, Germanic, and Slavic, without any explicit supervision regarding language genealogy.
Furthermore, these unsupervised vectors proved highly effective at predicting WALS features, particularly those related to syntax, outperforming previous methods that required parallel data.
Cross-lingual Performance
The utility of these embeddings was tested in zero-shot settings for two major tasks: dependency parsing and Natural Language Inference (NLI).
Dependency Parsing
In cross-lingual dependency parsing (Universal Dependencies v2.6), adding language embeddings resulted in an average improvement of 3.4 points in Unlabeled Attachment Score (UAS). The gains were particularly pronounced in Romance languages, with French seeing an improvement of over 10 points.
Natural Language Inference
For the XNLI task, integrating these embeddings into state-of-the-art multilingual models like XLM and XLM-R provided consistent performance boosts (up to 1.1 absolute points on average), demonstrating that structural language representations can enhance high-level semantic reasoning across languages.
Conclusion and Limitations
This work demonstrates that meaningful language representations can be learned without supervision. While highly effective for many language families, the model showed limitations with Semitic languages like Hebrew, likely due to challenges in unsupervised word alignment for those pairs. Future research will focus on expanding the diversity of the training set and integrating these embeddings into larger Transformer architectures to further bridge the gap for low-resource languages.
Summary updated