A dual-encoding system for dialect classification

Petru Rebeja, Dan CristeaCOLING 2020

TL;DR

This paper introduces a dual-encoding system for classifying Romanian dialects, combining TF-IDF features with a deep learning model. While the model shows promise on training data, it struggles with real-world evaluation data, highlighting the limitations of relying solely on explicit textual markers for dialect identification.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying Romanian dialects, specifically distinguishing between Moldavian and other Romanian idioms. Dialect identification is crucial for understanding linguistic variation and developing more nuanced natural language processing (NLP) tools. Previous work in dialect identification has explored various machine learning and deep learning approaches. This paper focuses on the VarDial evaluation campaign's Romanian Dialect Identification (RDI) task, aiming to build a system that can accurately classify text based on subtle differences in writing patterns characteristic of each dialect.

Methods

The core of the proposed system is a dual-encoding approach. For each input sentence, two separate TF-IDF encodings are generated: one tailored to Romanian dialect patterns and another to Moldavian. These two encodings are then concatenated and fed into a deep learning model. This deep learning component consists of a one-dimensional convolutional layer to identify prominent features, followed by a Global Max-Pooling layer. The extracted features are then processed through a series of Dense and Dropout layers with ReLU activation, culminating in an output layer for label prediction. The authors emphasize that their approach differs from some previous methods by focusing on shallow representations and giving more weight to learning distinctions between these representations for classification.

Datasets

The primary dataset used is the MOROCO dataset, which contains short text sentences from both Romanian and Moldavian dialects. This dataset also includes topic labels (culture, finance, politics, etc.). For evaluation, a supplementary corpus, MOROCO Tweets, was used, consisting of tweets as data samples.

Key Insights and Ideas

  • The "aha moment" of the paper is the realization that explicit textual markers, while useful for training, are not sufficient for robust dialect classification in real-world scenarios. The model's performance degradation on the evaluation set suggests that not all samples exhibit these distinct markers, and sometimes samples labeled as one dialect may use characteristics of another.
  • The intuition behind the dual-encoding TF-IDF approach is sound but limited. The idea is to capture specific writing patterns (e.g., the use of 'â' vs. 'î' in words like "atât"/"atît"). However, the evaluation data revealed that these patterns are not consistently present or are blurred, making discrimination difficult.
  • Overfitting is a significant concern. The authors observed that as they increased model complexity (more layers), training and validation accuracy improved, but evaluation accuracy decreased, indicating the model was learning to perform well on the training data but failing to generalize.
  • Reproducibility is addressed through Docker. The authors highlight the importance of creating a consistent and reproducible environment for machine learning experiments, especially when dealing with hardware constraints and development/training machine differences, by leveraging Docker.

Key Results

The paper reports results from two submitted runs to the RDI task.

  • First Submission:
    • Training Accuracy: 0.8091
    • Validation Accuracy: 0.7988
    • Evaluation Accuracy: 0.5553
  • Second Submission (with more layers):
    • Training Accuracy: 0.8220
    • Validation Accuracy: 0.8106
    • Evaluation Accuracy: 0.5402

The results clearly show a significant drop in performance from validation to evaluation sets, indicating overfitting. The second submission, despite higher training and validation scores, saw a slight decrease in evaluation accuracy, further confirming the overfitting issue. Confusion matrices reveal that the model heavily favors the Romanian dialect in the second submission, failing to detect Moldavian peculiarities.

Limitations and Future Work

The authors acknowledge several limitations:

  • Over-reliance on textual marks: The core design decision to discriminate primarily based on specific textual marks proved insufficient for the evaluation set, which contained fewer such distinctive samples and instances where dialectal writing norms were not strictly followed.
  • Overfitting: The model exhibits clear signs of overfitting, with performance degrading on unseen data as model complexity increases.
  • Hardware constraints: Limited computational resources led to a lack of extensive hyper-parameter tuning, forcing the authors to settle for a small set of variations. This also contributed to long training sessions.

Future work suggested includes:

  • Investigating the root cause of long training times to improve efficiency and enable more thorough hyper-parameter tuning.
  • Exploring additional signals beyond simple textual markings to improve classification accuracy.
  • Rewriting the whole model architecture if the investigation into training time reveals no single bottleneck.
  • Performing a proper hyper-parameter tuning once training efficiency is improved.

Summary updated

BibTeX
@inproceedings{rebeja2020dual,
  author = {Petru Rebeja and Dan Cristea},
  title = {{A dual-encoding system for dialect classification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure