Improving Cuneiform Language Identification with BERT

Gabriel Bernier-Colborne, Cyril Goutte, Serge LégerNAACL 2019

TL;DR

Researchers have developed a BERT-based model that achieves state-of-the-art performance in identifying cuneiform language varieties. This work demonstrates the effectiveness of large pre-trained language models for historical scripts and opens new avenues for analyzing ancient texts.

What Problem Does This Paper Address?

The paper tackles the challenge of Cuneiform Language Identification (CLI). Cuneiform, an ancient writing system, was used across various languages and dialects in the ancient Near East. Accurately identifying these distinct language varieties from short text segments is crucial for historical and linguistic research. The CLI shared task specifically aimed to distinguish between Sumerian and six dialects of Akkadian (Old Babylonian, Middle Babylonian peripheral, Standard Babylonian, Neo Babylonian, Late Babylonian, and Neo Assyrian). A key difficulty is the short length of many cuneiform text segments, making traditional language identification methods struggle.

Methods

The authors explored three main approaches:

  • SVM-based Baseline: A standard pipeline using Support Vector Machines (SVMs) trained on character n-grams (sequences of 1 to 4 characters). This is a common and effective method for language identification.
  • Voting Ensemble: An ensemble of multiple classifiers, including SVMs and probabilistic classifiers, combined using plurality voting. This approach aims to leverage the diverse strengths of different models.
  • BERT Approach: A deep learning model based on the BERT (Bidirectional Encoder Representations from Transformers) architecture. This involved:
  • Pre-training: The BERT model was first pre-trained on unlabeled cuneiform text using two tasks: Masked Language Modeling (MLM) to predict masked characters based on context, and Sentence Pair Classification (SPC) to determine if two segments belong to the same language. This phase was adapted to the specific characteristics of the cuneiform dataset. A second pre-training phase incorporated unlabeled test data.
  • Fine-tuning: The pre-trained model was then fine-tuned on the labeled CLI training data to predict the specific language variety.

Datasets

The primary dataset used was the Cuneiform Language Identification dataset created for the 2019 VarDial evaluation campaign. This dataset was built from the Open Richly Annotated Cuneiform Corpus.

  • Training Set: Unbalanced, with varying numbers of segments per class.
  • Development Set: Balanced, with an equal number of segments per class.
  • Test Set: Not explicitly described in detail, but assumed to be balanced.

The authors also performed deduplication on the training and development sets, noting that this significantly reduced the number of unique segments.

Key Insights and Ideas

  • BERT's Power for Historical Scripts: The paper demonstrates that large, pre-trained language models like BERT, originally developed for modern languages, can be effectively adapted to identify ancient scripts and their variations, even with limited labeled data.
  • Pre-training is Crucial: The two-phase pre-training strategy, especially the inclusion of unlabeled test data in the second phase, significantly boosted performance. This suggests that unsupervised learning on the target domain data is highly beneficial.
  • Short Text Challenge: The inherent difficulty of identifying languages from very short text segments remains a significant challenge, even for advanced models.
  • Deduplication Trade-off: While deduplicating the training data improved cross-validation estimates, its impact on the final test set performance was less pronounced, highlighting a potential trade-off between data cleanliness and data volume.

Key Results

  • The BERT-based model achieved state-of-the-art performance, reaching an accuracy of 77.11% and an F1-score of 0.7695 on the CLI test set.
  • This BERT model outperformed the voting ensemble by an absolute gain of approximately 2.5 F-score points.
  • The single SVM baseline (char[1-4]) achieved an accuracy of 74.53% and an F1-score of 0.7414.
  • The BERT model showed strong performance across most classes, with notable gains on the Old Babylonian (OLB) dialect.
  • Ablation studies indicated that the second pre-training phase (MLM only on training and dev data) provided the biggest performance gains for BERT.

Limitations and Future Work

  • Short Text Length: The paper acknowledges that the short length of text segments is a major source of errors, with most errors occurring on texts with 9 characters or less.
  • Label Ambiguity: A small percentage of texts in the dev set exhibited label ambiguity, where a text might have multiple plausible labels.
  • Future Work:
    • Modeling Symbol Shape: Exploring ways to model the visual shape of cuneiform symbols or their transliterations to better capture similarities.
  • Cross-Dataset Evaluation: Testing BERT on other language identification datasets.
  • Multi-label Classification: Casting the problem as a multi-label classification task, which might be more suitable for discriminating similar languages.

Summary updated

BibTeX
@inproceedings{berniercolborne2019improving,
  author = {Gabriel Bernier-Colborne and Cyril Goutte and Serge Léger},
  title = {{Improving Cuneiform Language Identification with BERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure