Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts

Ehsan Doostmohammadi, Minoo NassajianNAACL 2019

TL;DR

This paper tackles the challenging task of identifying languages and dialects in ancient cuneiform texts, a domain plagued by limited resources and complex tokenization issues. The authors explore various machine learning models, finding that an ensemble of Support Vector Machines and Naive Bayes, using character-level features, achieves the best performance with a macro-averaged F1-score of 72.10%.

What Problem Does This Paper Address?

Identifying the language or dialect of a text is a fundamental task in Natural Language Processing (NLP), crucial for applications like machine translation, information retrieval, and building language resources. However, this task becomes significantly more difficult when dealing with ancient scripts like cuneiform. Cuneiform texts present unique challenges:

  • Limited Resources: Unlike modern languages, cuneiform languages (Sumerian and various Akkadian dialects) have very few digitized and annotated resources available for training NLP models.
  • Tokenization Difficulties: Cuneiform writing is a complex system that can be both syllabic and logographic. There's no universal method or tool for breaking down cuneiform texts into meaningful units (tokens), making traditional word-level NLP approaches problematic.

The Cuneiform Language Identification (CLI) task in VarDial 2019 specifically aimed to address this by identifying seven languages/dialects: Sumerian and six Akkadian dialects (Old Babylonian, Middle Babylonian Peripheral, Standard Babylonian, Neo-Babylonian, Late Babylonian, and Neo-Assyrian).

Methods

The researchers investigated a range of machine learning methods, all relying on character-level features due to the aforementioned tokenization challenges. They employed both traditional machine learning algorithms and deep learning approaches:

Traditional ML Models (using Scikit-learn): * Support Vector Machine (SVM): A specific configuration with a low learning rate, hinge loss, and elasticnet penalty. * Naive Bayes: A multinomial Naive Bayes classifier. * Ensemble of SVM and Naive Bayes: A soft voting classifier combining the predictions of the SVM and Naive Bayes models. * Random Forest: A classifier with a specified number of estimators and depth. * Logistic Regression: A logistic regression classifier with a specific optimizer and training epochs.

These models used character n-grams (1-3 grams) with TF-IDF and character counts as input features.

Deep Learning Models (using TensorFlow): * Convolutional Neural Network (CNN): Employed parallel CNN layers with character embeddings as input. * Recurrent Neural Network (RNN): Utilized Long Short-Term Memory (LSTM) cells with character embeddings.

The deep learning models used character embeddings of size 32 for the 256 most common characters and were trained with an Adam optimizer.

Datasets

The primary dataset used was from the CLI shared task in VarDial 2019. This dataset comprises 7 classes: Sumerian (SUX) and six Akkadian dialects.

  • Training Data: 139,421 samples.
  • Development Set: 668 samples per label.
  • Test Set: 985 samples per label.

The paper notes that the training data is highly imbalanced, with Sumerian (SUX) and Neo-Assyrian (NEA) making up a significant portion (86.35%) of the data, while other dialects have much smaller sample sizes.

Key Insights and Ideas

  • Character-level features are sufficient: The study demonstrates that even without explicit word segmentation, character-level features can yield surprisingly good results for language and dialect identification in cuneiform texts. This is a crucial finding given the inherent difficulties in tokenizing these ancient scripts.
  • Ensemble methods shine: The combination of SVM and Naive Bayes (ensemble) outperformed individual models, highlighting the benefit of leveraging multiple learning strategies.
  • Deep learning struggles with limited data: While deep learning models (CNN, RNN) are powerful, they performed poorly on this task compared to traditional methods. This suggests that deep learning models require larger datasets than what was available for these low-resource cuneiform languages to effectively learn complex patterns.
  • Data imbalance is a significant challenge: The performance on certain dialects is notably lower, likely due to their sparse representation in the training data.

Key Results

  • The ensemble of SVM and Naive Bayes achieved the best performance on the development set with a macro-averaged F1-score of 72.10% and an accuracy of 73.56%.
  • The ensemble model also performed best on the test set, achieving a macro-averaged F1-score of 0.7210 and an accuracy of 0.7239.
  • The Naive Bayes classifier was the fastest to train, while the CNN and RNN were the slowest.
  • Late Babylonian (LTB) was the easiest class to identify, with a recall of 96.55%. Middle Babylonian Peripheral (MPB) and Old Babylonian (OLB) also showed relatively high recall.
  • Sumerian was often misclassified as Standard Babylonian, and Neo-Babylonian as Standard Babylonian. Neo-Assyrian was frequently misclassified as Neo-Babylonian or Standard Babylonian.

Limitations and Future Work

  • Low performance on some dialects: The authors acknowledge that the best model still struggles to accurately classify certain dialects, indicating a need for more sophisticated approaches.
  • Need for richer features: The paper suggests that incorporating word-level features (if tokenization can be improved) or leveraging embedded or transferred knowledge from related languages could significantly boost performance.
  • Hyperparameter tuning: The authors mention that further tuning of hyperparameters for deep learning models might improve their results, especially given the limited data.
  • Exploring other deep learning architectures: Future work could explore different deep learning architectures or pre-trained models that are more robust to low-resource scenarios.

Summary updated

BibTeX
@inproceedings{doostmohammadi2019investigating,
  author = {Ehsan Doostmohammadi and Minoo Nassajian},
  title = {{Investigating Machine Learning Methods for Language and Dialect Identification of Cuneiform Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure