Towards identifying the optimal datasize for lexically-based Bayesian inference of linguistic phylogenies

Taraka Rama, Søren WichmannCOLING 2018

TL;DR

This paper investigates the optimal number of meanings needed for accurate Bayesian phylogenetic inference of language families. They found that while more data generally improves results, there's a point of diminishing returns, and the optimal size is strongly correlated with the number of languages being studied, suggesting a rule of thumb of ~33 cognate classes per language.

What Problem Does This Paper Address?

Reconstructing the evolutionary history of languages, known as linguistic phylogenetics, often relies on comparing word lists across languages. Specifically, Bayesian phylogenetic methods, which are powerful tools borrowed from computational biology, require "cognate matrices." These matrices represent whether words for a given meaning in different languages are related (cognates) or not. Traditionally, researchers have used fixed sets of meanings, often around 100-200, based on Swadesh lists. However, there's been no systematic study to determine if this datasize is optimal, or if using more or fewer meanings impacts the accuracy of the inferred language family trees. This paper aims to fill that gap by empirically investigating how the number of meanings affects the quality of Bayesian phylogenetic inference.

Methods

The core of this research involves inferring phylogenetic trees using Bayesian methods on datasets of varying sizes. They tackle this by:

  • Ranking Meanings by Stability: They first rank meanings based on their "stability," which is operationalized by the number of cognate classes. Meanings with fewer cognate classes are considered more stable, implying less frequent lexical replacement.
  • Cumulative Datasets: They then create cumulative datasets, starting with the most stable meaning, then the two most stable, and so on, up to the total number of meanings available for each language family.
  • Bayesian Phylogenetic Inference: For each cumulative dataset, they run a Bayesian phylogenetic inference procedure using MrBayes. This involves constructing binary matrices from cognate judgments and applying standard models (Generalized Time Reversible Model with a discrete Gamma distribution for rate variation across sites).
  • Tree Evaluation: To assess the quality of the inferred trees, they employ two methods:
  • Comparison with Gold Standard Trees: For language families with well-established expert-derived trees, they calculate the Generalized Quartet Distance (GQD) between the inferred tree and the gold standard tree. The GQD measures the difference in tree topologies.
  • Comparison Between Successive Trees: For families where expert trees are less reliable or absent, they compute the GQD between trees inferred from successive meaning list sizes (e.g., the tree from n meanings vs. the tree from n+1 meanings). The assumption is that the point where this distance stabilizes indicates that adding more data no longer significantly improves the tree.

Datasets

The study utilizes data from eleven language families or subfamilies. For the primary analysis comparing against gold standard trees, they focus on five families:

  • Austronesian
  • Mayan
  • Mixe-Zoque
  • Indo-European
  • Uto-Aztecan

For the second evaluation method (comparing successive trees), they also include data from:

  • Afro-Asiatic
  • Bai
  • Chinese
  • Mon-Khmer
  • Ob-Ugrian
  • Tujia

The number of languages and meanings for each dataset is provided in Table 1. The data consists of multilingual word lists with expert-annotated cognate judgments.

Key Insights and Ideas

  • Stability Matters: The order in which meanings are added to the dataset significantly impacts tree quality. More stable meanings (those less prone to lexical replacement) are more informative for phylogenetic inference.
  • Diminishing Returns: Adding more meanings doesn't always lead to better trees. Beyond a certain point, the improvement plateaus, and in some cases (like Mayan), adding less stable, potentially loanword-influenced meanings can even degrade tree quality.
  • Proxy for Gold Standard: The method of comparing successive trees (without a gold standard) yields results remarkably similar to comparing against gold standard trees. This is a crucial finding, as it allows for the analysis of more language families where expert trees are unavailable.
  • Cognate Classes as a Better Metric: The number of cognate classes, rather than just the number of meanings, is a stronger predictor of optimal datasize. This suggests that the quality of the lexical data (how well it reflects shared inheritance) is more important than sheer quantity.

Key Results

  • For most families, the most significant improvement in tree quality (measured by GQD) occurs within the first 50-100 meanings.
  • The optimal number of meanings is strongly correlated with the number of languages in the family.
  • A proposed rule of thumb for optimal datasize is approximately 33 cognate classes multiplied by the number of languages being studied. This estimate is derived from both the gold standard and successive tree comparison methods.
  • For the gold standard method, the optimal number of cognate classes correlated with the number of languages with R² = 0.923 (p = 0.0094).
  • For the successive tree method, the correlation between the number of languages and cognate classes was R² = 0.876 (p < 0.0001).

Limitations and Future Work

  • Arbitrary Ties: When multiple meanings have the same stability rank, they are arbitrarily ordered, which could introduce minor variations.
  • Loanword Influence: The study acknowledges that loanwords can negatively impact tree inference, especially when less stable meanings are included. This is particularly noted for the Mayan family.
  • Future Work: The authors suggest that future research could be directed at extending phylogenies by adding data for lists that might not suffice on their own, implying a need for careful consideration of cognate class counts when expanding datasets. The developed rule of thumb provides a solid starting point for designing future lexical datasets for phylogenetic studies.

Summary updated

BibTeX
@inproceedings{rama2018towards,
  author = {Taraka Rama and Søren Wichmann},
  title = {{Towards identifying the optimal datasize for lexically-based Bayesian inference of linguistic phylogenies}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure