A Deep Generative Approach to Native Language Identification

Ehsan Lotfi, Ilia Markov, Walter DaelemansCOLING 2020

TL;DR

This paper introduces a novel approach to Native Language Identification (NLI) by fine-tuning a pre-trained generative language model (GPT-2) for each native language (L1). By assigning an unseen text to the L1 whose corresponding fine-tuned model yields the lowest language modeling loss, the method achieves state-of-the-art results, outperforming traditional machine learning and other deep learning approaches.

What Problem Does This Paper Address?

Native Language Identification (NLI) is the task of determining a writer's native language (L1) based on their writing in a second language (L2). This has applications in areas like security and marketing. Traditionally, NLI has been treated as a multi-class classification problem, relying heavily on handcrafted features designed to capture linguistic interference patterns from the L1 in the L2 writing. While these methods have achieved high accuracy (80-90%), they require significant feature engineering. More recent deep learning approaches, like CNNs and LSTMs, have shown promise but haven't consistently surpassed traditional SVMs with engineered features. This paper aims to leverage the power of large generative language models, specifically GPT-2, to achieve superior NLI performance without the need for manual feature extraction.

Methods

The core of this paper's contribution lies in its novel application of a generative language model, GPT-2, for classification. Instead of training a classifier from scratch, the authors fine-tune a pre-trained GPT-2 model separately for each target native language (L1). This means if they are identifying 11 L1s, they will have 11 fine-tuned GPT-2 models.

When presented with an unseen text, this text is fed into each of the fine-tuned L1 models. The language modeling loss (a measure of how well the model predicts the text) is calculated for each model. The authors hypothesize that the model fine-tuned on texts from the same L1 as the unseen text will exhibit the lowest language modeling loss. Therefore, the unseen text is assigned the L1 label corresponding to the model that produced the minimum loss. This approach is simple, avoids feature engineering, and leverages the generative capabilities of GPT-2 to implicitly capture L1-specific writing characteristics.

Datasets

The authors evaluate their approach on two commonly used NLI datasets:

  • TOEFL11: This corpus contains 12,100 essays in English from authors with 11 different native languages (Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, Turkish). The essays were written in response to eight prompts and vary in author proficiency. It's a benchmark dataset used in previous NLI shared tasks.
  • ICLE (ICLE-NLI subset): This dataset consists of essays from highly proficient non-native English speakers. For comparability, a 7-language subset was used, containing 110 essays per language (Bulgarian, Chinese, Czech, French, Japanese, Russian, Spanish), totaling 770 essays.

Key Insights and Ideas

  • Generative Models as Classifiers: The paper demonstrates that large generative language models, when fine-tuned on specific language groups, can be effectively repurposed for classification tasks like NLI, achieving state-of-the-art results.
  • Language Modeling Loss as a Discriminator: The core insight is that the language modeling loss itself can serve as a powerful signal for distinguishing between native languages. The model that "understands" the text best (lowest loss) is likely the one trained on that native language's writing patterns.
  • Implicit Feature Extraction: By fine-tuning GPT-2, the model implicitly learns the subtle linguistic interference cues characteristic of each L1, eliminating the need for manual feature engineering.
  • L1-Specific Fine-tuning is Key: The success hinges on creating individual models for each L1, allowing them to specialize in the unique patterns of that language group's writing in L2.

Key Results

The proposed GPT-2 based approach significantly outperforms previous state-of-the-art methods on both the TOEFL11 and ICLE datasets.

  • TOEFL11 Test Set: Achieved 89.0% accuracy, surpassing the previous best SVM (88.6%) and BERT (80.8%).
  • TOEFL11 10-fold Cross-Validation: Achieved 86.8% accuracy, matching the previous best SVM (86.4%) and outperforming BERT (76.3%).
  • ICLE 5-fold Cross-Validation: Achieved 94.2% accuracy, outperforming the previous best SVM (93.4%) and BERT (76.8%).

The confusion matrices reveal that German is the easiest L1 to identify, while Hindi, Telugu, Korean, and Japanese are the most challenging, consistent with previous findings.

Limitations and Future Work

  • Text Length Dependency: The authors hypothesize that the approach requires sufficiently long texts for the LM loss differences to become significant. Shorter texts might not provide enough signal.
  • Generalizability: While the approach is presented as generic for style-based classification, its effectiveness on other tasks and datasets needs further testing.
  • Future Work: The authors plan to test this hypothesis on other style-based classification tasks and explore its applicability to tasks requiring longer test samples for significant LM loss differences.

Summary updated

BibTeX
@inproceedings{lotfi2020deep,
  author = {Ehsan Lotfi and Ilia Markov and Walter Daelemans},
  title = {{A Deep Generative Approach to Native Language Identification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure