Extracting the Native Language Signal for Second Language Acquisition

Ben Swanson, Eugene CharniakNAACL 2013

TL;DR

This paper introduces a novel method for extracting linguistic patterns that are unique to a speaker's native language (L1) from their second language (L2) writing. This is crucial for Second Language Acquisition (SLA) research, as it allows for the creation of targeted educational materials by identifying specific L1-influenced linguistic habits.

What Problem Does This Paper Address?

Native Language Identification (NLI) is the task of determining a writer's L1 from their L2 text. While NLI has applications in areas like authorship attribution, its true value for SLA lies in identifying how an L1 influences L2 writing. Simply knowing a student's L1 is often trivial; what's needed is a way to pinpoint the specific linguistic patterns (e.g., grammatical structures, word choices) that students with a particular L1 tend to use unusually often in their L2. Existing NLI methods often struggle to identify these subtle, rare features due to biases in datasets and feature selection techniques that favor common patterns. This paper aims to bridge this gap by developing a robust method for extracting these discriminative, L1-specific linguistic features for SLA applications.

Methods

The core of the paper's contribution lies in its feature extraction and selection methodology. They utilize Tree Substitution Grammars (TSGs) as their primary feature type, arguing that TSG rules can capture complex syntactic structures more effectively than simpler features like n-grams or POS tags.

To address the challenge of identifying rare but discriminative features, they extend the standard Bayesian grammar induction algorithm. Their key innovation is a supervised mixture of latent grammars. This approach allows them to model multiple grammars simultaneously, each potentially capturing different aspects of the L1 signal. This is achieved by incorporating a Hierarchical Dirichlet Process (HDP), which enables the model to learn a distribution over grammars for each L1 and dataset combination.

For feature selection, they employ a two-stage process:

  1. Dataset Independence: They filter out features that are too strongly tied to specific corpora or essay prompts, ensuring the extracted patterns are generalizable. This is done using a $\chi^2$ test with a strict threshold to avoid discarding potentially valuable rare features.
  2. Relevancy and Redundancy: They evaluate several relevancy metrics, finding that the $\chi^2$ statistic is better suited than Information Gain (IG) or Symmetric Uncertainty (SU) for identifying rare, discriminative features. They also implement a redundancy detection mechanism, using Normalized Pointwise Mutual Information (NPMI) and a structural overlap metric, to remove redundant features and ensure a compact, informative set.

Datasets

The authors use four corpora for their experiments:

  • International Corpus of Learner English (ICLE): A common benchmark in NLI.
  • International Corpus of Crosslinguistic Interlanguage (ICCI): Annotated with sentence boundaries, previously unused in NLI.
  • Cambridge International Corpus (FCE): A public sample of prompted responses.
  • Lang8: A dataset of user-submitted text from an internet forum, noted for its noise and lack of constraints.

They specifically mention that they are releasing a list of L1-associated TSG rules derived from these corpora for public use.

Key Insights and Ideas

  • Mutual Information is Not Always Best: Standard relevancy metrics like Information Gain can be inappropriate for NLI tasks involving rare features, as they tend to favor more common patterns. The $\chi^2$ statistic proves more effective in identifying features that are truly discriminative of L1.
  • The Power of Rare Features: The most valuable signals for SLA often come from subtle, infrequently used linguistic patterns that are characteristic of a specific L1. The proposed method is designed to uncover these.
  • Supervised Mixtures for Granularity: Using a supervised mixture of latent grammars allows the model to learn more specific, fine-grained patterns associated with L1s, rather than forcing a single grammar to capture everything. This is analogous to breaking down a complex problem into smaller, more manageable sub-problems.
  • Dataset Independence is Crucial: Features that are artifacts of specific corpora (e.g., due to essay prompts) are not useful for general SLA. Robust filtering is needed to ensure extracted patterns reflect genuine L1 influence.
  • TSGs Capture Richness: Tree Substitution Grammars offer a powerful and flexible way to represent complex syntactic phenomena, making them ideal features for capturing nuanced L1 influences.

Key Results

  • The proposed supervised mixture of latent grammars (model M3) significantly improves the percentage of rules that reject the hypothesis of L1 independence, especially at stricter statistical significance levels, compared to baseline models. For example, at p < .01, M3 achieves 57.2% rule rejection compared to M1's 54.5%.
  • The $\chi^2$ statistic, when used as a relevancy metric, shows a better performance curve (lower expected loss) for the top features compared to IG and SU.
  • The structural overlap heuristic and SU perform similarly well for redundancy filtering, outperforming NPMI. They choose SU for its speed and generalizability.
  • The method successfully extracts a list of discriminative TSG rules, which they believe can be used for qualitative analysis and potentially integrated into SLA software.

Limitations and Future Work

  • Manual Interpretation: While the method extracts discriminative features, assigning specific linguistic causes to these patterns still requires manual qualitative analysis by experts.
  • Downstream Systems: Further work is needed to integrate these features into downstream automatic systems for SLA.
  • Data Expansion: The authors emphasize the need for continued incorporation of more data and induction of TSGs over larger datasets to improve the quality and scope of extracted patterns.
  • Language Transfer Phenomena: Future work could focus on determining the actual language transfer phenomena evidenced by the extracted patterns and distinguishing between preference and dispreference.
  • Automatic Integration: A key future direction is the automatic integration of these features into L1-targeted SLA software.

Summary updated

BibTeX
@inproceedings{swanson2013extracting,
  author = {Ben Swanson and Eugene Charniak},
  title = {{Extracting the Native Language Signal for Second Language Acquisition}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure