Advancing Linguistic Features and Insights by Label-informed Feature Grouping: An Exploration in the Context of Native Language Identification

Serhiy Bykh, Detmar MeurersCOLING 2016

TL;DR

This paper introduces a novel "label-informed feature grouping" technique for Native Language Identification (NLI). By clustering linguistic features (specifically verb subcategorization patterns) based on how they correlate with a writer's native language (L1), the method not only improves classification accuracy but also provides deeper insights into language learning phenomena.

What Problem Does This Paper Address?

Native Language Identification (NLI) aims to determine a writer's L1 based on their text written in a second language. While NLI has seen significant progress with sophisticated machine learning models and extensive feature sets, a major challenge remains: interpreting why these models work and what linguistic insights they offer, particularly for Second Language Acquisition (SLA) research. Current approaches often rely on large, complex feature sets that are difficult to analyze qualitatively. This makes it hard to understand the specific linguistic patterns that distinguish speakers of different L1s and to draw meaningful conclusions about language learning. The paper addresses this by proposing a method to group features in a way that explicitly leverages the L1 label, aiming to reveal underlying linguistic generalizations and improve interpretability.

Methods

The core contribution is a label-informed feature grouping technique. This method employs hierarchical clustering to group linguistic features. The key innovation is that the clustering process is informed by the classification label (the L1 of the writer).

Here's how it works:

  1. Variationist Perspective: The approach is grounded in variationist sociolinguistics, which studies linguistic choices made by speakers. The idea is that different L1s might lead learners to prefer certain linguistic variants.
  2. Feature Representation: The paper focuses on verb subcategorization (subcat) patterns as features. For each verb lemma (e.g., "believe"), they identify the different ways its arguments can be realized (e.g., "believe_sbj", "believe_sbj_obj").
  3. Label-Informed Clustering: Instead of creating a single feature vector for each verb, they create k vectors, where k is the number of L1 labels. Each vector represents the frequency proportions of subcat variants for a specific L1. Hierarchical clustering then groups verb lemmas that exhibit similar variant realization patterns across different L1s.
  4. Feature Reduction: Before clustering, several feature reduction steps are applied to mitigate data sparsity and noise. This includes ignoring argument permutations within a subcat variant and then grouping argument labels into broader categories (subject 's', object 'o', and others 'x'). Features for verb lemmas that only occur with a single subcat variant are also dropped.
  5. Evaluation: The resulting clusters of verb lemmas are then used as complex features in a classification setup. Different cut-offs of the dendrogram (representing different levels of grouping) are systematically evaluated to find the optimal grouping.

Datasets

The primary dataset used is the TOEFL11 corpus. This corpus was originally introduced for the NLI Shared Task 2013 and contains essays from English learners with 11 different L1 backgrounds (Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish). Each L1 is represented by 1,100 essays (900 for training, 100 for development, and 100 for testing). The paper uses the union of the training and development sets for training and the standard test set for testing.

Key Insights and Ideas

  • Grouping for Insight: The primary goal isn't just accuracy, but understanding why certain features are indicative of an L1. Grouping features based on L1-specific patterns allows researchers to move beyond individual feature occurrences to identify broader linguistic generalizations.
  • L1-Specific Preferences: The method confirms the hypothesis that learners with different L1s do indeed prefer different subcategorization patterns for verbs. This is a core tenet of SLA research.
  • Beyond Surface Features: While surface features (like word n-grams) are important, incorporating deeper linguistic structures (like subcat patterns) and grouping them intelligently can significantly improve NLI performance.
  • Qualitative Exploration: The technique enables fine-grained qualitative analysis. For example, it can help identify specific groups of verbs where a particular L1 shows a distinct preference, even if that preference isn't universal across all verbs.
  • Efficiency through Generalization: Grouping features can lead to more compact and efficient models by abstracting away from individual, potentially rare, verb lemmas to more general classes.

Key Results

  • Accuracy Improvement: While the basic setting with simple features achieved 44.5% accuracy, the best complex feature setting (with a cut-off of 0.3) reached 44.2%. However, when combined with a binary verb lemma model (restoring surface information), the accuracy significantly improved. The best model ([s/c, +bvm] with cut-off 0.7) achieved 48.0% accuracy, outperforming the simple feature-only model by 1.8%.
  • State-of-the-Art Performance: When integrated into an ensemble with other feature types, the proposed method contributed to achieving an overall accuracy of 85.4% on the TOEFL11 dataset, slightly outperforming the previously reported best accuracy of 85.3%.
  • Qualitative Findings: The qualitative exploration revealed that while overall patterns for subject usage (pronoun vs. noun) might not be strongly indicative of L1 Chinese, specific verb groups do show indicative preferences, supporting the hypothesis that Chinese learners might overuse pronoun subjects.

Limitations and Future Work

  • Feature Reduction Trade-offs: The paper acknowledges that feature reduction, while necessary for sparsity, leads to a loss of potentially indicative subcat information. A more fine-grained reduction is suggested as future work.
  • Data Sparsity: While grouping helps with sparsity, the paper notes that using more data or more frequent variables could further increase quantitative gains.
  • Generalizability: The authors plan to explore applying the label-informed feature grouping technique to other NLP tasks beyond NLI to assess its broader applicability.
  • Syntactic and Morphological Features: Future work will also involve exploring other linguistic feature types (syntactic and morphological) within the variationist framework.
  • Link to SLA Theory: A key future direction is to establish a firmer link between the data-induced patterns and established SLA theories of L1-transfer.

Summary updated

BibTeX
@inproceedings{bykh2016advancing,
  author = {Serhiy Bykh and Detmar Meurers},
  title = {{Advancing Linguistic Features and Insights by Label-informed Feature Grouping: An Exploration in the Context of Native Language Identification}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2016},
}
Figure