TL;DR
This paper presents a meta-classifier system for identifying languages and dialects written in ancient cuneiform script, a challenging task due to the script's age and limited digital resources. The system achieved a respectable 4th place in the Cuneiform Language Identification shared task, outperforming a neural network model and highlighting the continued effectiveness of n-gram-based approaches for this domain.
What Problem Does This Paper Address?
Language identification (LID) is a well-established field, but it typically focuses on contemporary languages. Identifying historical languages, especially those written in ancient scripts like cuneiform, presents unique challenges. Cuneiform, one of the earliest writing systems, was used for over three millennia, evolving into various languages and dialects. Accurately distinguishing between these historical linguistic variations using computational methods is crucial for digital humanities and historical linguistics, but it's a relatively underexplored area. This paper tackles the specific problem of identifying Sumerian and Akkadian (with its six dialects) from cuneiform texts.
Methods
The authors developed a system based on a meta-classifier. This approach combines the predictions of multiple individual classifiers to make a final decision, often leading to improved robustness and performance. Specifically, their meta-classifier was trained on the outputs of several Support Vector Machine (SVM) models. These SVMs were trained using character n-gram and character skip-gram features. These features capture patterns in the sequence of characters within the text, which are indicative of language. The meta-classifier essentially learns how to best combine the "votes" from these individual SVMs. This is an adaptation of previous successful meta-classifier systems used in other VarDial shared tasks.
Datasets
The primary dataset used for this work was provided by the Cuneiform Language Identification (CLI) shared task, organized as part of the VarDial Evaluation Campaign 2019. This dataset, described in detail by Jauhiainen et al. (2019), consists of transliterated cuneiform texts. It includes two languages: Sumerian and Akkadian, with Akkadian further divided into six dialects: Old Babylonian, Middle Babylonian peripheral, Standard Babylonian, Neo Babylonian, Late Babylonian, and Neo Assyrian. Table 1 in the paper provides a breakdown of the number of texts, lines, and signs for each language/dialect.
Key Insights and Ideas
- N-grams still reign supreme (for now): Despite the rise of deep learning, character n-gram-based meta-classifiers proved highly effective for cuneiform language identification, even outperforming a sophisticated compositional recurrent neural network (RNN) in this specific task.
- Historical LID is a distinct challenge: The performance gap between the top systems and others, and the comparison with the neural model, suggests that historical language identification, particularly with ancient scripts, requires specialized approaches.
- Meta-classifiers offer a robust ensemble: The success of the meta-classifier demonstrates the power of combining multiple simpler models to achieve superior results, especially when dealing with complex linguistic variations.
- The value of shared tasks: The CLI shared task provided a much-needed benchmark and dataset for this niche area, fostering research and enabling direct comparisons between different LID approaches.
Key Results
The authors' submission, under the team name "PZ," achieved an F1 macro score of 0.738, ranking 4th out of eight teams in the CLI shared task. This performance was only a few percentage points behind the top three systems. Notably, their meta-classifier significantly outperformed a compositional RNN model (ranked 6th) submitted by the "ghpaetzold" team, achieving a higher F-score and better distinguishing between challenging language pairs like Neo Babylonian and Neo Assyrian.
Limitations and Future Work
The authors acknowledge that their system, while effective, was not the top-performing one. They express interest in analyzing the results of the highest-performing teams, particularly the top team which used a neural-based method. They hypothesize that the success of the top team might be attributed to more advanced deep learning techniques like BERT. Future work could involve exploring these advanced neural architectures for cuneiform LID and further investigating the linguistic features that make certain language pairs particularly difficult to distinguish.
Summary updated