TL;DR
This paper explores using Native Language Identification (NLI) models, typically used to guess a writer's first language, to automatically detect "collocation transfer" – when a learner uses word combinations common in their native language in their second language. They found that NLI features can indeed reveal these transfer patterns, offering a broader net than traditional methods for identifying subtle L2 learning influences.
What Problem Does This Paper Address?
Learning a second language (L2) often involves "transfer," where a learner's first language (L1) influences their L2. This can be helpful, but "negative transfer" (interference) can lead to communication inefficiencies, like using awkward word pairings. Identifying these transfer patterns is crucial for improving L2 teaching and learning resources. Traditional methods for studying transfer require pre-defined linguistic structures and specific L1/L2 data, limiting their scope. This paper proposes using NLI, a machine learning task that predicts a writer's L1 based on their L2 text, as a more flexible tool to discover potential collocation transfer, even for less-studied languages and without prior assumptions about specific transfer phenomena.
Methods
The core of this research is applying NLI with a specific focus on collocations as features. Collocations are words that frequently appear together (e.g., "strong coffee," not "powerful coffee"). The authors chose collocations because they are easily interpretable and directly reflect psychological associations between words.
Their approach involved:
- Feature Engineering: They extracted four types of collocations: adverb-verb, three-word bundles with a verb, verb-noun, and adjective-noun pairs. Dependency parsing was used to ensure these were genuine collocations.
- Feature Filtering: To manage the large number of potential features and mitigate topic influence (where certain words might appear frequently simply because of the essay's topic), they employed two key filtering steps:
- Entropy-based filtering: Features with low entropy (meaning they appear in a limited range of prompts within an L1 group) were removed, as these are more likely to be topic-driven rather than L1 transfer.
- ANOVA test: This was used to ensure features were not just common within a specific L1 group but also showed differences across L1 groups, indicating potential L1 influence.
- Classification: A Ridge Classifier was used for NLI. This model was chosen for its ability to handle sparse data (common with lexical features), efficiency, and importantly, its coefficient values can highlight features that are highly characteristic of a specific L1, thus signaling potential transfer.
- Analysis: They used hierarchical clustering to group L1s based on their collocation patterns and confusion matrices to visualize misclassifications, aiming to link model performance to linguistic similarities between L1 groups.
Datasets
The primary dataset used is the International Corpus of Learner English (ICLE), a collection of college student essays. The authors filtered this corpus to include 15 L1s, resulting in approximately 5,600 pieces of writing. For a specific comparison with native English, they also incorporated the LOCNESS corpus.
Key Insights and Ideas
- NLI as a "Wide Net" for Transfer: The study successfully demonstrates that NLI models, by identifying characteristic features of L1 speakers, can indeed uncover potential collocation transfer patterns that might be missed by traditional, hypothesis-driven methods.
- Collocations as Powerful NLI Features: The research validates collocations as highly effective features for NLI, suggesting they capture significant L1 influence in L2 production.
- Topic Influence is a Real Challenge: The authors highlight the difficulty of disentangling genuine L1 transfer from topic-specific vocabulary, necessitating sophisticated filtering techniques like entropy-based removal.
- L1 Idiosyncrasies Matter: The performance of the NLI model varies significantly across L1s, with less unique (more similar) collocation patterns leading to higher misclassification rates. This suggests that the distinctiveness of an L1's collocation system plays a key role in how well it can be identified.
- Indirect Transfer is a Promising Area: Beyond direct transfer (e.g., using a specific L1 collocation), the paper hints that NLI might also reveal indirect transfer effects, such as avoidance patterns or influences on L2 perception, which warrants further investigation.
Key Results
- The NLI model achieved an accuracy of 61% using approximately 1,800 collocation features, significantly outperforming baseline models (random guessing, most frequent label, stratified) which achieved accuracies between 7% and 17%.
- The model identified features with high coefficients that aligned with previously identified collocation transfer instances for L1 French (verb bundles) and L1 Chinese (intensifier-verb collocations).
- For L1 Chinese, the high-coefficient features contained fewer intensifiers compared to native English writers, confirming previous findings.
- Hierarchical clustering revealed that L1 groups with similar collocation patterns (e.g., Norwegian, Swedish, Finnish, German) were harder for the model to distinguish, leading to higher misclassification rates between them.
- L1s with higher performance (e.g., Chinese, Japanese, Tswana, Italian) generally had more distinct collocation profiles.
Limitations and Future Work
- Data Sparsity: Lexical features can be sparse, meaning they might not appear in all test data, impacting model performance. Longer texts could improve this.
- Topic Influence Mitigation: While entropy filtering was used, the optimal threshold is not tested, and it's unclear if all topic-induced features were removed.
- Proficiency Levels: The ICLE corpus has unbalanced proficiency levels across L1 groups. Low proficiency might lead the classifier to pick features reflecting general L2 difficulty rather than specific L1 transfer.
- Classifier Performance: The confidence in identified transfer features is higher for L1s with good classification performance.
- Future Work: The authors suggest investigating indirect transfer effects revealed by NLI features and exploring the impact of different entropy thresholds and balanced proficiency levels. They also note the potential for using NLI to identify transfer in other linguistic domains beyond collocations.
Summary updated