Cross-corpus Native Language Identification via Statistical Embedding

Francisco Rangel, Paolo Rosso, Julian Brooke, Alexandra UitdenbogerdNAACL 2018

TL;DR

This paper introduces a novel statistical embedding method (LDSE) for Native Language Identification (NLI) that significantly outperforms traditional n-gram based approaches in cross-corpus scenarios. By representing documents based on the statistical distribution of term weights across different languages, LDSE achieves competitive results even with scarce and imbalanced data, offering a more robust solution for identifying a writer's native language from their L2 text.

What Problem Does This Paper Address?

Native Language Identification (NLI) is the task of determining a writer's native language (L1) based on their writing in a second language (L2). This has practical applications in education (providing targeted feedback), forensics, security, and marketing. A key challenge in NLI is the cross-corpus scenario: training a model on data from one source (e.g., social media) and evaluating it on data from a completely different source. This is particularly difficult because different corpora can have varying characteristics, data distributions, and even writing styles. Previous NLI shared tasks have shown that while n-gram based features combined with SVMs are effective, performance drops significantly in cross-corpus settings. The authors aim to develop a more robust NLI method that can generalize well across different corpora, even when data is limited and imbalanced.

Methods

The paper proposes a Low Dimensionality Statistical Embedding (LDSE) approach. Unlike traditional methods that rely on explicit linguistic features or n-grams, LDSE represents documents based on the statistical distribution of term weights across different native languages.

Here's a breakdown of the LDSE method:

  • Term Weighting: For each document, tf-idf weights are calculated for all terms.
  • Distributional Representation: The core idea is that a document written by a native speaker of language X should have a term weight distribution similar to other documents written by native speakers of X. LDSE captures this by calculating various statistical measures of these term weight distributions for each document.
  • Feature Extraction: For each document, the LDSE representation is a vector composed of:
  • Average and median values of term weights.
  • Minimum and maximum values of term weights.
  • First and third quartiles of the weight distribution.
  • Gini indexes (to measure skewness and kurtosis).
  • The first nine moments of the distribution.
  • Dimensionality Reduction: Crucially, this method results in a low-dimensional representation (17 features per class) compared to high-dimensional n-gram spaces, making it more efficient and potentially more robust to noise.
  • Machine Learning: Standard machine learning algorithms (Naive Bayes, Simple Logistic, SVM, Neural Networks) are then applied to these LDSE features for classification.

The authors highlight that LDSE does not require external resources, linguistic tools, or complex preprocessing, making it a self-contained and efficient approach.

Datasets

The authors utilize several publicly available corpora for their experiments:

  • Lang8: A corpus of learner English texts, used for training and testing.
  • TOEFL: A corpus of learner English texts, often used for NLI research.
  • ICLE (International Corpus of Learner English): Another corpus of learner English.
  • ICNALE (International Corpus of Native and Non-native Arabic Learner English): Specifically includes Arabic learner data.
  • EFCamDat (EF Cambridge Open Language Database): Used in prior work, but not directly for training in this paper's main experiments.
  • SWT (Student Writing Task) Corpus: A newly introduced benchmark for their case study, containing essays from Computer Science PhD students in Australia. This corpus is characterized by its small size, imbalanced native languages, and a single topic.

The paper notes that the Indonesian data in the ICNALE corpus is particularly scarce.

Key Insights and Ideas

  • Statistical Distributions as Powerful Features: The most significant insight is that the statistical distribution of term weights within a document, when compared across different native languages, provides a highly discriminative signal for NLI. This moves beyond simple frequency counts of words or characters.
  • Cross-Corpus Robustness: LDSE demonstrates remarkable performance in cross-corpus scenarios, outperforming traditional n-gram methods. This suggests that the statistical properties captured by LDSE are more invariant to corpus-specific variations than surface-level features.
  • Low Dimensionality Advantage: The LDSE representation significantly reduces dimensionality while retaining discriminative power. This is crucial for handling imbalanced datasets and potentially for computational efficiency.
  • Competitive Performance with Scarce Data: The method shows competitive results even when evaluated on very small and imbalanced datasets like the Indonesian portion of ICNALE, highlighting its resilience.

Key Results

The paper reports significant improvements of LDSE over baseline character and word n-gram models across various cross-corpus setups.

  • Macro-averaged F1-score Improvements: LDSE consistently outperformed the baseline n-gram models, with improvements ranging from 4.39% to 19.27%.
  • Best Performance: The highest improvements were observed for Arabic, while the best absolute results were achieved for Indonesian.
  • SWT Case Study: In the Australian academic scenario (SWT corpus), LDSE achieved results comparable to those on Lang8 and TOEFL, demonstrating its effectiveness in a more controlled, academic setting.
  • Baseline Performance: The baseline n-gram models achieved macro-averaged F1-scores around 50-70% in cross-corpus settings, which LDSE significantly surpassed.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the ICNALE corpus, particularly for Indonesian, is very small, leading to potential variability in results.
  • While the authors focus on identifying Arabic, Chinese, and Indonesian, the current LDSE formulation is "one-vs-all," and extending it to a true multi-class scenario across many languages might present challenges in aligning distributions.
  • Future Work:
    • The authors explicitly state their aim to address NLI from a multi-class perspective in future work.
  • Investigating the impact of different statistical features and moments on performance.
  • Exploring the application of LDSE to other languages and corpora.
  • Further analysis of why LDSE is more robust to cross-corpus variations.

Summary updated

BibTeX
@inproceedings{rangel2018cross,
  author = {Francisco Rangel and Paolo Rosso and Julian Brooke and Alexandra Uitdenbogerd},
  title = {{Cross-corpus Native Language Identification via Statistical Embedding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure