Language Discrimination and Transfer Learning for Similar Languages: Experiments with Feature Combinations and Adaptation

Nianheng Wu, Eric DeMattos, Kwok Him So, Pin-zhen Chen, Çağrı ÇöltekinNAACL 2019

TL;DR

This paper explores language and dialect identification for closely related languages using Support Vector Machines (SVMs) with various feature combinations and an adaptation technique. The authors demonstrate that simple linear SVMs with carefully chosen features can achieve state-of-the-art results, even outperforming more complex models in some cases, and highlight the effectiveness of adaptation for improving performance, especially when test data differs significantly from training data.

What Problem Does This Paper Address?

Distinguishing between closely related languages or dialects of the same language is a notoriously difficult task in Natural Language Processing (NLP). While standard language identification (LI) focuses on clearly distinct languages, this paper tackles the more nuanced challenge of identifying subtle differences in written text, where linguistic variations might be minimal. This is crucial for applications like targeted information retrieval, content moderation, and understanding regional linguistic diversity. The paper also investigates cross-lingual transfer learning, where knowledge gained from one language variety is applied to another, a particularly relevant problem when dealing with low-resource languages or dialects.

Methods

The core of the proposed approach relies on Support Vector Machines (SVMs), a well-established machine learning algorithm for classification. The authors experimented with two main strategies for combining features:

  • Flat Feature Combination: This involves concatenating various n-gram features (character and word n-grams of different lengths) into a single, large feature vector. This approach aims to capture a broad spectrum of linguistic signals.
  • SVM Ensembles: This method involves training multiple SVM classifiers, each using a different set of features or configurations, and then combining their predictions. The paper specifically mentions using a "mean probability rule" for fusion.

A key innovation explored is an adaptation method. This technique involves retraining the SVM model by incorporating high-confidence predictions from the development or test set into the training data. This is particularly useful when the test data distribution is expected to diverge from the training data, allowing the model to "learn on the fly."

Datasets

The paper evaluates their methods on several datasets from the VarDial 2019 Evaluation Campaign:

  • Cuneiform Language Identification (CLI): A dataset with seven related languages within a wide historical timeframe, presented as Unicode transcriptions.
  • Discriminating between Mainland and Taiwan variation of Mandarin Chinese (DMT): Two tracks (traditional and simplified script) for distinguishing between Putonghua and Guoyu.
  • German Dialect Identification (GDI): A dataset based on Swiss German dialects around Bern, including text transcripts and i-vectors (acoustic features).
  • Moldavian vs. Romanian Cross-dialect Topic Identification (MRC): This task involved binary classification between Moldavian and Romanian, and also cross-lingual topic classification.

Key Insights and Ideas

  • Simplicity Can Be Powerful: The study consistently shows that simple linear SVMs with well-chosen n-gram features can be highly competitive, even outperforming more complex models in certain scenarios. This challenges the notion that deep learning is always necessary for such tasks.
  • Feature Engineering Matters: The choice and combination of n-gram features (character and word) significantly impact performance. The paper suggests that a broad range of features, especially character n-grams, are crucial for capturing subtle linguistic differences.
  • Adaptation is Key for Divergent Data: The adaptation technique proved particularly effective when the test data distribution differed from the training data. This highlights the importance of flexible models that can adjust to new data characteristics.
  • Traditional Chinese Script Advantage: For the Mandarin dialect identification task, the traditional script consistently yielded better results, suggesting it carries more discriminative information than the simplified script.
  • Speaker Variation is a Challenge: In German Dialect Identification, the adaptation method's effectiveness was particularly pronounced, suggesting that speaker-specific variations, beyond just dialectal differences, play a significant role and can be mitigated through adaptation.

Key Results

  • The team achieved first place in German Dialect Identification (GDI) and MRC subtasks 2 and 3.
  • They secured second place in the simplified DMT task and CLI.
  • In most tasks, the flat combination of features performed better than SVM ensembles.
  • For the GDI task, the flat SVM model with character n-grams (1-5), word unigrams/bigrams, and i-vectors achieved an F1-score of 75.93, closely followed by other top entries.
  • The adaptation method showed significant improvements, especially in the GDI and CLI tasks, with some F1-scores increasing by up to 30% in MRC subtask 2 after adaptation.

Limitations and Future Work

  • Limited Comparison of Adaptation Methods: While adaptation was shown to be effective, the paper notes that their official submissions didn't always include direct comparisons with and without adaptation, making a definitive conclusion about its utility in all cases difficult.
  • Ensemble Performance: While flat combinations generally outperformed ensembles, the authors acknowledge that their ensemble approach was not exhaustively explored, and further work could investigate different combination methods and architectures.
  • MRC Subtask Ineffectiveness: The ineffectiveness of adaptation on MRC subtasks 2 and 3 was unexpected, and the authors could not offer a definitive explanation.
  • Future Work: The authors plan to extend their work on ensemble models to cover more diverse combination methods and architectures. They also aim to further investigate the impact of adaptation and its potential pitfalls.

Summary updated

BibTeX
@inproceedings{wu2019language,
  author = {Nianheng Wu and Eric DeMattos and Kwok Him So and Pin-zhen Chen and Çağrı Çöltekin},
  title = {{Language Discrimination and Transfer Learning for Similar Languages: Experiments with Feature Combinations and Adaptation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure