TL;DR
This paper investigates the robustness of compression-based language complexity metrics by applying them to a diverse set of South American indigenous languages. The findings largely validate these metrics, suggesting their universality, and reveal a fascinating link between morphological/syntactic complexity and language phylogeny.
What Problem Does This Paper Address?
Quantifying "language complexity" is a crucial but challenging endeavor in linguistics and Natural Language Processing (NLP). While intuitive, formal definitions are elusive. Existing complexity metrics, often developed with a focus on Indo-European languages, need validation across a broader linguistic landscape to ensure their universality. This is particularly important for developing NLP tools that are equitable and effective for all languages, not just the most widely studied ones. The paper aims to bridge this gap by testing established complexity metrics on a rich dataset of South American indigenous languages, exploring whether their proposed properties hold true.
Methods
The core methodology involves applying a set of compression-based language complexity metrics, derived from information theory, to textual data. These metrics quantify complexity by measuring how much a text's size changes when compressed, with different metrics targeting morphological, syntactic, and pragmatic aspects of language.
Specifically, the paper uses:
- Degradation-based metrics: These metrics work by "degenerating" (e.g., deleting or replacing) specific linguistic units (characters for morphology, words for syntax, sentences for pragmatics) and then measuring the impact on text compressibility. A higher impact (less compressibility) indicates higher complexity for that linguistic tier.
- Information Theory: The underlying principle is that the complexity of a message (text) can be approximated by the size of its compressed representation.
- Statistical Analysis: The researchers analyze correlations and variances of the computed complexity scores across different languages and linguistic tiers to validate pre-defined hypotheses.
Datasets
The primary dataset used is a collection of New Testament translations into 51 South American indigenous languages, kindly provided by IBM Research Brazil. This dataset is a subset of a larger corpus originally assembled for language classification and machine translation research.
For comparison and validation, the study also includes 5 Indo-European languages (English, French, German, Portuguese, Spanish) and Ancient Greek.
The paper notes that the Bible was chosen due to the scarcity of parallel texts across many indigenous languages and its historical role in their documentation. However, it acknowledges potential biases in these translations, as they may have been produced with limited regard for cultural nuances and may not be representative of the languages' full complexity.
Key Insights and Ideas
- Universality of Metrics: The study largely validates the robustness of compression-based complexity metrics across a diverse set of South American indigenous languages, suggesting these metrics can indeed capture universal linguistic properties.
- Phylogeny Matters: A significant finding is the strong correlation between language phylogeny (family tree) and the trade-off between morphological and syntactic complexity. This suggests that these complexity measures can be useful for distinguishing between languages.
- South American Languages are Diverse: The study highlights that South American languages exhibit greater dispersion in their complexity measures compared to Indo-European languages, reinforcing the need for broader linguistic coverage in complexity research.
- Orthographic Complexity as a Confound: The paper identifies orthographic complexity as a potential confounding factor, particularly when comparing languages with vastly different writing systems (e.g., Hebrew vs. Ancient Greek), impacting the validation of the "original language" hypothesis.
Key Results
- Hypothesis H2 (Equi-complexity): Confirmed. The variance in compressed text sizes and overall complexity scores was significantly smaller than the variance in original text sizes, supporting the idea that complexity measures are relatively stable across languages.
- Hypothesis H3 (Trade-off): Confirmed. Negative correlations were consistently found between syntactic and morphological complexity, indicating a trade-off between these two linguistic aspects. This trade-off was observed to be dependent on language phylogeny.
- Observation O1: Confirmed. Positive correlations were observed between morphological complexity and the number of unique word types, and negative correlations with the number of word tokens.
- Observation O2: Confirmed. The variance in pragmatic complexity was significantly lower than that of morphological and syntactic complexities, suggesting a more universal aspect to inter-sentential information transmission.
- Hypothesis H1 (Original Language Complexity): Inconclusive. Ancient Greek was not consistently found to be the least complex language, potentially due to orthographic differences and unknown translation histories.
- Observation O3 (Nichols' Metric): Inconclusive due to a very small sample size of languages for which comparable data was available.
Limitations and Future Work
- Author Language Expertise: The authors acknowledge their primary expertise in Portuguese, English, and Spanish, limiting their ability to provide deep insights into the nuances of the indigenous languages studied.
- Translation Bias: The use of Bible translations, while practical, introduces potential biases and may not fully represent the natural complexity of these languages. The history of these translations is also largely unknown, impacting the interpretation of results for Hypothesis H1.
- Writing System Dependency: The current metrics are strongly tied to alphabetic and low-logographic writing systems. Future work should extend these metrics to other writing systems.
- Future Research: The authors propose further investigation into the inconclusive propositions, particularly focusing on the impact of orthographic complexity and exploring a wider range of writing systems. They also aim to obtain data on translation histories to better interpret results related to Hypothesis H1.
Summary updated