TL;DR
This paper introduces an algorithm that automatically identifies cognates (words with a common origin) between related languages using statistical analysis of character correspondences. This method offers a computationally efficient way to aid historical linguists in reconstructing language relationships, especially when dealing with large datasets or complex sound changes.
What Problem Does This Paper Address?
Identifying cognates is a fundamental task in historical linguistics, crucial for understanding language evolution and relationships. Traditionally, this process relies heavily on expert human judgment, which can be time-consuming and subjective, especially when comparing numerous languages or dealing with extensive word lists. The challenge lies in developing an objective, systematic, and computationally efficient method to detect these shared linguistic roots, even when sound correspondences are not simple one-to-one mappings. The authors aim to automate this process, providing a tool that can rank word pairs by their probability of being cognates.
Methods
The core of the algorithm is a statistical approach to identifying sound correspondences. It takes two wordlists (preferably in phonemic transcription, but not necessarily) as input. The process involves:
- Initial Character Correspondence: For each word pair, it considers all possible character-to-character mappings.
- Observed vs. Expected Frequencies: It constructs an "TxS matrix" (where T and S are the sizes of the alphabets in the two languages) to compare the observed frequencies of character pairs with their expected frequencies under a random distribution.
- Weighting Correspondences: The difference between observed and expected frequencies is used to assign a "weight" to each possible character correspondence.
- Potential Calculation: A "potential" is calculated for each correspondence, incorporating its weight and the potentials of subsequent possible correspondences. This helps in identifying the most likely sequence of correspondences.
- Null Correspondences: The algorithm accounts for cases where characters in one language might not have a direct equivalent in the other (e.g., deletion).
- Iterative Refinement: The process can be iterated, recomputing weights and potentials based on refined positional matching, further improving accuracy.
- Cognation Index: A "cognation index" is derived by dividing the maximum potential of a word pair by its number of correspondences, providing a measure of how likely the pair is to be cognate.
- Vowel/Consonant Disallowance: A significant improvement comes from disallowing correspondences between vowels and consonants, which is handled by an auxiliary algorithm.
- Weight Adjustment: Further refinements involve adjusting weights by setting low positive weights to zero and handling character-to-zero matches, aiming to reduce noise and improve signal.
Datasets
The paper uses a hypothetical Austronesian-like language example for illustration. The primary evaluation dataset is a corpus of approximately 300 words across 75 languages and dialects of Vanuatu. This dataset is used to test the algorithm's performance in identifying cognates.
Key Insights and Ideas
- Statistical Power of Character Frequencies: The algorithm demonstrates that statistical analysis of character frequencies, even without prior knowledge of phonemic systems, can reveal significant patterns of correspondence indicative of cognation.
- Context-Free to Context-Sensitive: While the initial matching is context-free, the iterative refinement and the potential for identifying context-sensitive rules (mentioned in future work) show a path towards more sophisticated linguistic analysis.
- Importance of Vowel/Consonant Distinction: Disallowing vowel-consonant correspondences dramatically improves performance, highlighting the biological and functional differences between these sound classes.
- Cognation Index as a Robust Measure: The proposed "cognation index" provides a quantifiable metric for assessing the likelihood of a word pair being cognate, allowing for ranking and filtering.
- Handling Complex Sound Changes: The algorithm shows promise in identifying cognates even when sound changes are complex, though its performance degrades with very low cognate proportions.
Key Results
- The algorithm can process a 250-item bilingual list in about five seconds of CPU time on a DEC KL1091.
- It achieves excellent results for languages sharing 40% or more cognates, even with complex sound correspondences.
- Performance deteriorates when cognate proportions are lower and sound correspondences are complex, but remains excellent for predominantly one-to-one correspondences.
- In a test on Sakao and Tolomako (Vanuatu languages), the algorithm identified 38.91% cognates (with a cutoff index of 1.28), closely matching a human estimate of 41%.
- Out of 50 word pairs with the highest cognation indices, only two were definitively not cognate and one was doubtful, indicating high precision.
Limitations and Future Work
- Environment-Conditioned Correspondences: The paper identifies the identification of environment-conditioned phonological correspondences as the "next, most obvious stage" for improvement. This is challenging because the frequencies of correspondences in specific environments can become too low for statistical significance.
- Rules of Metathesis: Identifying rules of metathesis (transposition of sounds) is another desirable improvement, which the authors suggest is subordinated to the discovery of context-sensitive rules.
- Data Requirements: While the algorithm can handle non-phonemic transcriptions, its performance is likely better with more precise phonetic information. The paper notes that performance deteriorates with lower cognate proportions.
- Computational Cost of Advanced Approaches: Approaches inspired by chess-playing programs for handling complex environments have been tried but proved too expensive in computer time.
The authors envision the algorithm as a step towards a more general problem: given two texts, determining if one is a translation of the other and establishing translation rules.
Summary updated