Naive Bayes and BiLSTM Ensemble for Discriminating between Mainland and Taiwan Variation of Mandarin Chinese

Li Yang, Yang XiangNAACL 2019

TL;DR

This paper tackles the challenge of automatically distinguishing between Mandarin Chinese spoken in Mainland China and Taiwan, a task crucial for many NLP applications. The authors propose an ensemble system combining traditional Naive Bayes models with deep learning BiLSTM models, achieving strong performance on the DMT shared task. Their findings highlight the effectiveness of character trigrams for Naive Bayes and the robustness of an ensemble approach for this dialect identification task.

What Problem Does This Paper Address?

Dialect identification, a sub-field of language identification, aims to differentiate between variations of the same language. While language identification has seen significant progress, dialect identification remains a tougher challenge due to the subtler linguistic differences involved. This paper focuses on a specific instance: distinguishing between Mandarin Chinese spoken in Mainland China and Taiwan. These two varieties, while mutually intelligible, exhibit differences in character sets (simplified vs. traditional), vocabulary, and pronunciation. Accurately identifying these variations is important for applications like speech recognition, machine translation, and data acquisition, where understanding the origin of the text can significantly improve downstream performance. The DMT (Discriminating between Mainland and Taiwan Variation of Mandarin Chinese) shared task at VarDial 2019 provided a benchmark for this specific problem.

Methods

The authors explore a range of methods, starting with traditional machine learning models and progressing to deep learning approaches, culminating in an ensemble system.

  • Traditional Machine Learning: They experiment with Logistic Regression (LR), Support Vector Machines (SVM), and Multinomial Naive Bayes (MNB). These models are trained on "bag of n-gram" features, which represent the text as a collection of sequences of n words or characters. They investigate the impact of different n-gram sizes (1 to 4) and combinations, focusing on character bigrams and trigrams as particularly effective.
  • Deep Learning Models: Recognizing the success of deep learning in NLP, they evaluate seven different architectures: CNN-based (CNN, DCNN, DPCNN), RNN-based (BiLSTM, Self-attentive BiLSTM), and CNN-RNN hybrids (CNN-BiLSTM, BiLSTM-CNN). These models utilize pre-trained word embeddings (like Word2vec) to represent words and then process these sequences to learn sentence representations. The core framework involves an input embedding layer, a sentence encoder layer (the specific deep learning model), and an output layer for classification.
  • Ensemble Methods: To further boost performance, they combine the predictions of multiple models using four strategies: Mean Probability, Highest Confidence, Majority Voting, and Meta-Classifier (Classifier Stacking). The goal is to leverage the strengths of different models to achieve a more robust and accurate prediction.

Datasets

The primary dataset used is the DMT dataset, provided for the VarDial 2019 evaluation campaign. This dataset consists of labeled sentences from news articles originating from Mainland China and Taiwan. It's notable that the organizers provided two versions of the dataset: one with simplified Chinese characters and another with traditional Chinese characters. This means that character form, a potential distinguishing feature, could not be used for discrimination in this task. The dataset is described as having a balanced class distribution, with 18,770 instances for training, 2,000 for validation, and 2,000 for testing.

Key Insights and Ideas

  • Character N-grams Shine for Naive Bayes: For the Naive Bayes classifier, character trigrams proved to be the most effective individual feature, slightly outperforming character bigrams. This suggests that local character patterns are highly indicative of dialectal variation.
  • MNB's Robustness: Multinomial Naive Bayes (MNB) consistently outperformed LR and SVM across various n-gram features, establishing itself as a very strong baseline for this task.
  • Deep Learning vs. Traditional Models: While deep learning models generally outperformed LR and SVM, they were often outperformed by MNB, indicating that for this specific task and dataset, carefully engineered n-gram features with a strong classifier could be more effective than complex deep learning architectures.
  • Ensemble Power: Combining models, particularly MNB and BiLSTM, through ensemble methods like Mean Probability or Highest Confidence, yielded the best overall performance, demonstrating the benefit of aggregating diverse predictive signals.
  • Traditional vs. Simplified: Performance was slightly better on the traditional Chinese character dataset compared to the simplified version, suggesting potential subtle differences in how these character sets affect feature extraction.

Key Results

The paper reports macro-weighted F1 scores for various experiments.

  • Best Individual Features: For MNB, character trigram achieved an F1 score of 0.9015 (simplified) and 0.9150 (traditional).
  • Best Combined Features: Combining character bigram and trigram features with MNB yielded F1 scores of 0.9080 (simplified) and 0.9225 (traditional).
  • Best Deep Learning Model: BiLSTM achieved the highest F1 scores among the deep learning models, with 0.9000 (simplified) and 0.9115 (traditional).
  • Best Ensemble Performance: The ensemble of MNB and BiLSTM using Mean Probability or Highest Confidence achieved the top results, reaching 0.9130 (simplified) and 0.9240 (traditional) on the validation set.
  • Shared Task Performance: The final submission (MNB + BiLSTM ensemble) achieved macro-weighted F1 scores of 0.8530 and 0.8687 on the test sets for the simplified and traditional versions, respectively.

Limitations and Future Work

  • Test Set Performance: The authors note that their models performed consistently lower on the test set compared to the validation set. They suggest that fine-tuning parameters with k-fold cross-validation or employing other overfitting prevention strategies could improve test set performance.
  • Limited Scope: The current work focuses solely on Mainland China and Taiwan variations. The authors express interest in extending their model to encompass a wider range of Mandarin Chinese varieties, such as those spoken in Hong Kong, Macao, Singapore, and Malaysia, to further examine its effectiveness.

Summary updated

BibTeX
@inproceedings{yang2019naive,
  author = {Li Yang and Yang Xiang},
  title = {{Naive Bayes and BiLSTM Ensemble for Discriminating between Mainland and Taiwan Variation of Mandarin Chinese}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure