TL;DR
This paper introduces sequence models for computationally understanding word borrowing between languages, tackling two key tasks: predicting how a word will be incorporated into a new language and identifying the origin of borrowed words. By framing etymology prediction as a sequence-to-sequence problem, they demonstrate that neural models, particularly Transformers, can achieve strong performance, offering valuable tools for lexicon expansion and language revitalization.
What Problem Does This Paper Address?
Languages are dynamic entities, constantly evolving through the adoption of words from other languages. This process, known as borrowing, is a fundamental aspect of linguistic change, yet computationally modeling it remains a significant challenge. Understanding how and from where words enter a language has implications for creating more comprehensive dictionaries, improving machine translation systems, and even aiding in language preservation. Existing work often focuses on cognates (words with shared origins) or transliteration, which don't fully capture the complexities of borrowing from unrelated languages, where words can be translated literally (calques), adapted phonetically, or semantically extended. This paper aims to bridge this gap by developing computational models for these diverse borrowing processes.
Methods
The authors frame computational etymology as a sequence-to-sequence modeling problem, leveraging neural networks. They propose two main tasks:
- Task 1: Incorporation Prediction: Given a donor word, its language, the target language, and the etymological relation (e.g., calque, semantic loan), predict the form of the incorporated word in the target language.
- Task 2: Donor Prediction: Given an incorporated word and its language, predict the donor language, donor word, and the etymological relation.
They experiment with several neural architectures, including:
- LSTM encoder-decoders: A standard recurrent neural network approach.
- LSTM with copy attention: An extension that allows the model to directly copy parts of the input sequence, useful for words with similar spellings.
- Transformers: A more recent architecture that uses attention mechanisms, proving highly effective in many NLP tasks.
- Ensemble methods: Combining the predictions of multiple models to improve overall performance.
They also explore variations like multi-task learning (training a single model on all borrowing relations) versus single-task learning (training separate models for each relation) and the use of SentencePiece for subword tokenization.
Datasets
The primary dataset used is derived from the English edition of Wiktionary, extracted using the Yawipa parser. This dataset contains over 150,000 annotated borrowing relationships across 837 languages. The authors focus on six specific types of borrowing relations: calques, partial calques, semantic loans, phono-semantic matches, transliterations, and a generic "bor" category. They note that the "bor" category constitutes the vast majority (96%) of the dataset, posing a challenge for modeling less frequent borrowing types.
Key Insights and Ideas
- Borrowing is more than just transliteration: Unlike cognates, borrowed words undergo diverse transformations (translation, semantic shifts, phonetic adaptation) that require more sophisticated modeling than simple character-level mapping.
- Multi-task learning is beneficial: Training a single model on all borrowing relations generally outperforms models trained separately for each relation, suggesting that these processes share underlying linguistic patterns.
- Transformers excel: The Transformer architecture, with its attention mechanism, achieves the best performance on Task 1, highlighting its ability to capture complex dependencies in linguistic data.
- Ensembling boosts performance: Combining predictions from multiple models, especially through score-based voting, significantly improves accuracy, particularly for Task 1.
- Low-resource languages are a key focus: The models demonstrate a capability to handle numerous low-resource languages, where traditional methods might struggle due to data scarcity.
- Task 2 is more tractable: Predicting the donor language and relation is generally easier than predicting the exact incorporated word form, with models achieving high accuracy in identifying these aspects.
Key Results
- Task 1 (Incorporation Prediction): The Transformer model achieved the highest character BLEU score (61.30) and accuracy (22.19). The ensemble method further improved accuracy to 25.67. However, models struggled significantly with less frequent borrowing types (partial calques, semantic loans, etc.), often defaulting to treating them as generic "bor" borrowings.
- Task 2 (Donor Prediction): LSTM models achieved high accuracy (around 96%) in predicting the donor language and relation. Word prediction accuracy was lower (around 47.9%), but character edit distance indicated that predictions were often close to the gold standard, suggesting useful outputs for lexicographers.
Limitations and Future Work
The authors acknowledge that the extreme imbalance in the dataset, with the "bor" category dominating, makes it difficult to accurately model less frequent borrowing types. This data sparsity is a major limitation, especially for separately trained models.
Future work could involve:
- Developing custom loss functions that provide "partial credit" for predictions that are close but not exact, especially for related languages or near misses.
- Investigating more sophisticated methods for handling low-resource languages and rare borrowing types.
- Exploring alternative architectures or training strategies to better capture the nuances of different borrowing processes.
- The authors also suggest that their models can be used to suggest possible etymologies and plausible unknown word forms, which can then be verified by human lexicographers, aiding in dictionary creation and language revitalization efforts.
Summary updated