Using Cross-Lingual Part of Speech Tagging for Partially Reconstructing the Classic Language Family Tree Model

Anat Samohi, Daniel Weisberg Mitelman, Kfir BarACL 2022

TL;DR

This paper proposes a novel method for understanding language relationships by using AI to predict Part-of-Speech (POS) tags across different languages. By measuring how well a POS tagger trained on one language performs on another, the researchers create a new "similarity graph" that surprisingly reconstructs significant portions of the traditional language family tree. This suggests that POS information, often overlooked in favor of lexical similarity, is a powerful signal for historical language connections.

What Problem Does This Paper Address?

The classic model of language evolution depicts languages as branching out from a common ancestor, forming a tree-like structure. While this "language family tree" is useful for understanding genetic relatedness, some researchers question its ability to capture the full picture of language proximity. Languages borrow words and are influenced by others in ways that aren't purely genetic. This paper tackles the challenge of creating a more nuanced measure of language similarity that goes beyond traditional lexical comparisons and can potentially reconstruct the classic tree model using a different type of linguistic information.

Methods

The core of this research lies in leveraging a pre-trained multilingual language model (mBERT) for a cross-lingual Part-of-Speech (POS) tagging task. The authors employ a "zero-shot" learning approach: they fine-tune mBERT to predict POS tags for texts in a "source" language and then evaluate its performance on texts from a "target" language. The resulting F1 score (a measure of accuracy) for each source-target language pair serves as a proxy for their similarity.

To reconstruct the language family tree, each language is represented by a vector of these F1 scores, capturing its similarity to all other languages based on POS tagging performance. These vectors are then clustered using the k-means algorithm. The authors compare the resulting clusters to the established language family tree, using the Wu-Palmer similarity metric (a standard way to measure tree distance) as a baseline for comparison.

Datasets

The study utilizes treebanks from Universal Dependencies (UD) for training and evaluation. The Universal POS (UPOS) tagset is used, which includes core POS categories applicable across many languages. The authors include 36 languages in their study, chosen to represent a diversity of language families and subfamilies, all of which are covered by mBERT.

Key Insights and Ideas

  • POS Tags as a Linguistic Fingerprint: The research demonstrates that Part-of-Speech tagging performance across languages can serve as a powerful indicator of their underlying relationships, even when they are not lexically similar or share the same script.
  • Cross-Lingual Transferability: Fine-tuned mBERT models show remarkable ability to generalize POS tagging knowledge across languages, suggesting that the model captures underlying linguistic structures that transcend specific languages.
  • Reconstructing the Tree: The POS-based similarity scores, when clustered, successfully reconstruct significant portions of the classic language family tree, validating the hypothesis that this type of linguistic information is highly relevant to historical language evolution.
  • Geographic Influence: The clustering of Baltic and Uralic languages, alongside geographically proximate countries, hints at a potential geographical dimension in the POS-based similarity method, which warrants further investigation.

Key Results

The study found that models trained on genetically related languages generally perform better on each other, even if they use different scripts (e.g., Spanish model performing well on Russian). For instance, a Spanish model showed high F1 scores when evaluated on English and Russian. The k-means clustering of the 36 languages resulted in 9 clusters, with 31 out of 36 languages being classified correctly according to the classic model. The correlation between their POS-based similarity (F1 scores) and the traditional Wu-Palmer similarity metric was found to be strong (Pearson=0.64, Spearman=0.59).

Limitations and Future Work

The authors acknowledge that the size and quality of the treebanks used can influence results. They also note that mBERT's pre-training on Wikipedia might introduce biases towards languages with larger Wikipedia collections. To mitigate this, they designed each language vector to include F1 scores from all other languages, ensuring a balanced representation.

Future work could involve exploring the discovered geographical dimension in language similarity and investigating how to further refine the model to account for more nuanced linguistic influences beyond POS tags. The authors also suggest that their approach could be extended to other NLP tasks for cross-lingual transfer.

Summary updated

BibTeX
@inproceedings{samohi2022using,
  author = {Anat Samohi and Daniel Weisberg Mitelman and Kfir Bar},
  title = {{Using Cross-Lingual Part of Speech Tagging for Partially Reconstructing the Classic Language Family Tree Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure