The R2I_LIS Team Proposes Majority Vote for VarDial’s MRC Task

Adrian-Gabriel ChifuNAACL 2019

TL;DR

This paper presents a simple yet effective majority vote ensemble model for discriminating between Moldavian and Romanian news texts. Despite using only handcrafted features and standard machine learning models, their approach achieved a strong second-place ranking in the VarDial 2019 evaluation. This highlights the power of well-chosen features and ensemble methods for dialect identification tasks.

What Problem Does This Paper Address?

Distinguishing between closely related language varieties, such as dialects, is a challenging task for natural language processing (NLP). This is particularly true for languages like Romanian and Moldavian, which are linguistically very similar, with differences often manifesting in subtle lexical and grammatical variations. The VarDial (NLP for Similar Languages, Varieties and Dialects) workshop series aims to address this by organizing shared tasks focused on computational analysis of such language variations. This paper tackles the Moldavian vs. Romanian Cross-dialect Topic identification (MRC) task, specifically the binary classification sub-task: determining whether a given text is written in Moldavian or Romanian.

Methods

The R2I_LIS team's approach is built on a straightforward yet robust methodology. They engineered a set of 40 handcrafted features, categorized into five types: token statistics, character-based, punctuation-based, word-based, and named entity-based. These features were designed to be simple to compute and understand, aiming to prove their effectiveness.

These features were then fed into five standard supervised machine learning models:

  • K-Nearest Neighbors (KNN)
  • Logistic Regression (LR)
  • Support Vector Machine (SVM)
  • Neural Network (NN)
  • Random Forest (RF)

The crucial element of their approach is the majority vote ensemble. Instead of relying on a single model, the final decision for classifying a text as Moldavian or Romanian is determined by the consensus of these five models. This ensemble strategy is intended to improve performance and stabilize the model's predictions.

Datasets

The primary dataset used for this task is the MOROCO dataset, specifically the portion designated for the binary classification sub-task of the Moldavian vs. Romanian Cross-dialect Topic identification (MRC) task. This dataset was collected from news domains and comprises training, development, and test sets. The paper notes that for the closed training task, only subsets of the provided training data were used, without external resources.

  • Training set: 21,719 samples
  • Development set: 11,845 samples
  • Test set: 5,923 samples

Key Insights and Ideas

  • Simplicity can be powerful: The paper strongly advocates for the effectiveness of simple, handcrafted features. Their success demonstrates that complex, deep learning architectures are not always necessary for achieving competitive results in dialect identification, especially when features are well-chosen.
  • Ensemble methods boost performance and robustness: The majority vote ensemble proved to be a key factor in their success. By combining the predictions of multiple diverse models, they were able to achieve better overall performance and a more stable classification, mitigating the weaknesses of individual models.
  • Feature normalization matters: One of their runs (Run3) achieved the best performance by normalizing certain features (character, punctuation, and word features) by dividing them by the total number of characters in the text. This suggests that scaling features appropriately can significantly impact model performance.
  • Training data size is important: Comparing their runs, the authors found that using both the "train" and "dev" sets for training (concatenated) generally led to better performance than using only the "train" set, highlighting the benefit of more training data.

Key Results

The R2I_LIS team submitted three runs to the VarDial 2019 MRC task. Their best performing run (Run3), which utilized normalized features and trained on concatenated "train" and "dev" data, achieved a macro-averaged F1-score of 0.7964 on the test set. This performance secured them the second-place ranking among all participants in the MRC task's first sub-task. Their other runs (Run1 and Run2) ranked third and fourth, respectively, with macro-F1 scores of 0.7781 and 0.7762 on the test set.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Hyperparameter tuning: The current hyperparameter settings were chosen empirically. A more rigorous cross-validation protocol is suggested for more robust model optimization.
  • Feature selection: The paper proposes applying feature selection methods to identify the most useful features among the 40 handcrafted ones, potentially leading to a more efficient and effective model.
  • Overfitting: While generally not a significant issue, the authors note that for the Neural Network models, there was a notable difference between train and test performance, suggesting potential overfitting in those specific cases.

Summary updated

BibTeX
@inproceedings{chifu2019r,
  author = {Adrian-Gabriel Chifu},
  title = {{The R2I\_LIS Team Proposes Majority Vote for VarDial’s MRC Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure