TwistBytes - Identification of Cuneiform Languages and German Dialects at VarDial 2019

Fernando Benites, Pius von Däniken, Mark CieliebakNAACL 2019

TL;DR

This paper presents TwistBytes' approaches for identifying German dialects and cuneiform languages at the VarDial 2019 workshop. They achieved third place in both tasks, demonstrating the effectiveness of their meta-classifier system combined with features like i-Vectors and language models.

What Problem Does This Paper Address?

The paper tackles two distinct but related challenges in natural language processing: identifying regional dialects within a language and distinguishing between ancient written languages.

For German Dialect Identification (GDI), the goal is to differentiate between four specific dialects of Swiss German (Bern, Basel, Lucerne, Zurich). This is particularly challenging because Swiss German lacks a standardized written form, leading to highly subjective transcriptions and variations in spelling even for the same word. This ambiguity can make it difficult for even native speakers to understand dialects from neighboring regions.

For Cuneiform Language Identification (CLI), the task is to classify texts written in cuneiform script into Sumerian or one of six Akkadian dialects (Old Babylonian, Middle Babylonian Peripheral, Standard Babylonian, Neo-Babylonian, Late Babylonian, and Neo-Assyrian). Cuneiform is one of the oldest known writing systems, used for millennia by different cultures in Mesopotamia, making its identification a complex linguistic and historical problem.

Methods

The TwistBytes team employed a meta-classifier approach for both tasks, building upon their previous work.

For GDI, their system, dubbed TB-Meta, uses a two-tier architecture. The first tier involves multiple "base classifiers," each trained on a specific feature set extracted from the input sentence. These feature sets include:

  • Textual features: Term Frequency (TF) and TF-Inverse Document Frequency (TF-IDF) for character and word n-grams (n=1 to 7).
  • i-Vectors: These are dense features derived from audio signals, provided by the organizers, which capture speaker characteristics and are known to be effective for dialect identification.

The outputs of these base classifiers are then combined using an "S-Classifier" (summing predictions), which serves as input to a final "meta-classifier" (a linear SVM). This meta-classifier makes the final prediction. A key innovation this year was the integration of semi-supervised learning, where high-confidence predictions on the unlabeled test set were used as additional training data to improve performance.

For CLI, the TB-LM system utilizes TF-IDF features on symbol level (n-grams from 1 to 3) and language modeling features. They train a 3-gram language model with Kneser-Ney smoothing for each language and use the resulting language model scores as additional features. A linear SVM is used as the classifier.

Datasets

  • GDI: The system was trained and evaluated on the ArchiMob data set, which contains transcribed sentences from four Swiss German dialect areas. The paper notes that the training set was slightly modified from the previous year, and the validation set was the test set from the 2018 GDI shared task.
  • CLI: The system was trained and evaluated on a dataset of cuneiform texts, provided by the organizers, covering Sumerian and six Akkadian dialects. The paper includes tables detailing the distribution of instances per language in the training, validation, and test sets.

Key Insights and Ideas

  • i-Vectors are powerful for dialect identification: The paper strongly suggests that i-Vectors, even when used cautiously due to potential speaker-specific issues, significantly boost performance in dialect identification tasks, especially when dealing with limited speaker data.
  • Meta-classification and feature fusion are effective: The TB-Meta architecture, which combines predictions from multiple base classifiers trained on diverse features, proved highly effective, outperforming simpler SVM models.
  • Semi-supervised learning can enhance performance with limited labeled data: The integration of semi-supervised learning, particularly for GDI, showed a marginal but notable improvement, highlighting its utility when labeled data is scarce or when dealing with out-of-domain data.
  • Language modeling is crucial for ancient scripts: For CLI, the use of language model perplexity scores as features, alongside TF-IDF, was instrumental in distinguishing between closely related ancient languages and dialects.
  • Ambiguity in transcriptions is a major hurdle: The paper emphasizes the challenges posed by subjective transcriptions and the lack of standardized orthography in Swiss German, making character-based and error-tolerant methods essential.

Key Results

  • GDI: The TwistBytes system (TB-Meta-SiV) achieved 3rd place out of 6 teams with a macro-averaged F-1 score of 74.55% on the test set. Their validation set performance was remarkably high (0.9031), suggesting the validation set might have been easier to classify than the test set.
  • CLI: The TwistBytes system (TB-LM) achieved 3rd place out of 8 teams with a macro-averaged F-1 score of 0.74 on the test set. The TB-LM system performed slightly better on the test set (0.74) than on the validation set (0.73).

Limitations and Future Work

  • GDI Test Set Performance Drop: The authors note a significant drop in performance on the test set compared to the validation set for their best GDI system (TB-Meta-SiV). They suspect differences in data curation between the sets and intend to investigate this further.
  • Speaker Identification Sensitivity: While i-Vectors were beneficial, the authors acknowledge a potential risk: if i-Vectors for a specific speaker are misidentified, it could negatively impact the classifier's performance.
  • CLI Language Identification Challenges: For CLI, the paper highlights that NEB and OLB were the hardest dialects to identify. While OLB had fewer training samples, the authors point out that sample size alone isn't the sole determinant of distinguishability.
  • Future Work: The paper doesn't explicitly detail future work beyond investigating the GDI test set performance discrepancy. However, the success of i-Vectors and semi-supervised learning suggests these avenues could be further explored and refined. The CLI task also presents opportunities for more advanced language modeling techniques or exploring symbol-level features more deeply.

Summary updated

BibTeX
@inproceedings{benites2019twistbytes,
  author = {Fernando Benites and Pius von Däniken and Mark Cieliebak},
  title = {{TwistBytes - Identification of Cuneiform Languages and German Dialects at VarDial 2019}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure