TL;DR
This paper introduces the SIGTYP 2022 Shared Task, a challenge to predict cognate reflexes (words with a common origin) in different languages using computational methods. The task highlights the regularity of sound changes in language evolution and showcases how advanced machine learning, particularly convolutional neural networks, can achieve impressive results in this linguistic domain.
What Problem Does This Paper Address?
Historical linguists often group words from related languages into "cognate sets," where each word is a "cognate reflex" or "reflex." These reflexes typically exhibit regular sound correspondences, meaning a specific sound in one language consistently maps to another sound in a related language (e.g., English 't' often corresponds to German 'z' in words like "ten" vs. "zehn"). This regularity allows linguists to predict how a word might sound in one language if its cognate exists in another.
The challenge, however, is to automate this prediction process. While linguists can leverage extensive knowledge of multiple language pairs, computational systems need to learn these complex sound correspondences from data. This paper addresses the problem of building computational models that can predict missing cognate reflexes in a language, given the reflexes of the same cognate set in other languages. This is crucial for tasks like reconstructing ancestral languages and understanding language evolution.
Methods
The SIGTYP 2022 Shared Task focused on reflex prediction, where systems were tasked with predicting missing word forms within cognate sets. The core methodology involved:
- Data Preparation: Standardized multilingual wordlists from various language families were used. For the task, some word forms were artificially deleted from these lists to create "missing" reflexes that systems had to predict.
- Evaluation Metrics: Performance was measured using several metrics:
- Edit Distance (ED): The number of operations (insertions, deletions, substitutions) needed to transform a predicted word into the actual word.
- Normalized Edit Distance (NED): Edit distance normalized by string length to account for varying word lengths.
- B-Cubed F-Scores: A metric that emphasizes the regularity of predictions, penalizing systematic errors less than edit distance.
- BLEU Scores: Adapted from machine translation, this metric assesses the similarity between predicted and actual word forms.
- Baselines: Two baseline systems were established:
- A CORPAR (Correspondence Pattern Recognition) based system.
- A Support Vector Machine (SVM) based system, which generally outperformed the CORPAR baseline.
- System Architectures: Participating teams employed a range of approaches, including:
- Neural Networks: Notably, convolutional neural networks (CNNs) and transformer-based models.
- Phylogenetic Inference: One system utilized Bayesian phylogenetic inference to incorporate evolutionary relationships.
- Phonetic Alignment: Several systems relied on phonetic alignment techniques, similar to the extended baselines.
Datasets
The study utilized datasets from the Lexibank repository, which hosts wordlists from over 100 standardized datasets. These datasets were curated and made available with standardized phonetic transcriptions and, where available, expert-identified cognates.
- Training Data: Datasets with identifiers preceded by an asterisk were automatically searched for cognates.
- Surprise Data: Datasets without asterisks provided expert cognates.
- Benchmark Creation: For the shared task, the researchers created artificial test sets by deleting words from existing cognate sets. This allowed for a controlled evaluation of prediction accuracy. The data was split into five partitions, with test sets containing 10%, 20%, 30%, 40%, and 50% of the original words, respectively.
Key Insights and Ideas
- Regularity of Sound Change: The task's success, even with limited data, underscores the remarkable regularity of sound changes in language evolution. This regularity is the bedrock upon which computational prediction can be built.
- Neural Networks Excel: Machine learning models, particularly those based on neural networks like convolutional neural networks (CNNs) and transformers, demonstrated superior performance, especially in more challenging scenarios with less training data.
- B-Cubed F-Scores Highlight Systematicity: The B-Cubed F-score proved valuable in evaluating prediction quality by emphasizing systematicity. This metric revealed that even systems with some individual errors could perform well if their sound correspondences were consistent.
- Targeted vs. General Approaches: While neural networks generally led, targeted approaches (like the JULIA system from CrossLingference) that incorporated linguistic knowledge (e.g., phylogenetic information) or specialized alignment techniques performed competitively, sometimes even outperforming neural systems.
- Data Sparsity is a Challenge: The paper acknowledges that data for historical linguistics can be sparse and patchy, meaning not all cognates are represented across all languages or have survived without significant changes. This poses a fundamental challenge for any computational approach.
Key Results
The Mockingbird team's system "I1" (an image inpainting model using CNNs) emerged as the overall winner, consistently achieving the best performance across most evaluation metrics and dataset proportions.
- The Mockingbird "I1" system was ranked first in the aggregated ranks for normalized edit distance, B-Cubed F-Scores, and BLEU scores.
- Other Mockingbird systems (N1-A, N1-B, N1-C) also performed very well, securing top positions.
- The SVM baseline showed surprisingly strong performance, especially with smaller test sets (10% retained data).
- The CrossLingference "JULIA" system demonstrated strong performance on B-Cubed F-Scores, particularly when training data was limited, highlighting its ability to capture systematic patterns.
- Generally, performance tended to decrease as the proportion of data retained for testing increased, indicating that systems were more effective at predicting when more data was missing.
Limitations and Future Work
- Limitations:
- Evaluation of Lost Words: Predictions for words that have been completely lost from a language cannot be directly evaluated, requiring artificial data creation.
- Data Sparsity and Patchiness: The inherent limitations of historical linguistic data (missing reflexes, irregular sound changes) pose a significant challenge.
- Replicability: Due to time constraints, the organizers could not individually verify the replicability of all submitted systems.
- Future Work:
- The paper suggests that predicted proto-forms could be used by scholars to manually correct them, or by field linguists to guide elicitation of specific lexical items.
- The success of the shared task highlights the potential for computational methods to assist in various annotation tasks in historical linguistics, such as phylogenetic reconstruction and sound correspondence detection.
- The authors aim to encourage further research by making the data, code, and results publicly available.
Summary updated