A Memory-Sensitive Classification Model of Errors in Early Second Language Learning

Brendan Tomoschuk, Jarrett LovelettNAACL 2018

TL;DR

This paper introduces a novel approach to predicting errors in early second language learning by leveraging linguistic and cognitive features, including memory-related ones. Their random forest model significantly outperforms a baseline, demonstrating the power of memory dynamics and user-specific patterns in understanding language acquisition.

What Problem Does This Paper Address?

Learning a new language is a complex process influenced by numerous factors, from the inherent properties of the languages involved to the learner's cognitive abilities and study habits. While research has explored various linguistic and cognitive influences on vocabulary acquisition, there's a need for predictive models that can identify potential learning difficulties early on. This paper aims to address this by building a model that predicts errors in early second language learners using data from the Duolingo app, focusing on understanding the interplay of linguistic features and memory-related cognitive factors.

Methods

The authors employed a random forest classifier for their predictive modeling. This choice was motivated by the algorithm's scalability to large datasets, its resistance to overfitting, and its minimal need for parameter tuning. The core innovation lies in their feature engineering. Instead of relying solely on standard linguistic features, they incorporated memory-related features derived from the Duolingo usage data. Notably, they replaced traditional dummy-encoded categorical features for "user" and "token" with mean and variance of error rates for those users and tokens. This memory-efficient approach captures individual learning patterns and token difficulty without the computational burden of high-dimensional categorical representations. They also engineered several interaction terms between linguistic and memory features to capture more nuanced relationships.

Datasets

The study utilized data collected from Duolingo in 2017, specifically focusing on learners in their first 30 days of using the app. The dataset comprised three learner populations: English speakers learning Spanish and French, and Spanish speakers learning English. The data was split into training (80%), development (10%), and test (10%) sets. The training data contained approximately 1.88 million exercises, with similar proportions for the development and test sets.

Key Insights and Ideas

  • Memory Dynamics are Crucial: The most impactful features for predicting errors were the mean and variance of error rates for each user. This highlights that a learner's past performance and the consistency of their errors are highly predictive of future difficulties.
  • User History Matters More Than Identity: Replacing user IDs with aggregated error statistics (mean and variance) was a computationally efficient way to capture individual learning trajectories and significantly improved model performance.
  • Token Difficulty is Multifaceted: Similarly, using mean and variance of error rates for tokens, combined with features like part-of-speech, effectively captured token-specific difficulty beyond simple frequency.
  • Spacing Effects are Predictive: Various lag features, representing the time elapsed between learning instances of words or their stems, consistently ranked high in importance, underscoring the well-established benefits of spaced repetition for memory retention.
  • Engineered Features Outperform Raw Data: The authors' engineered features, particularly those related to memory and user/token error aggregation, were critical for the model's success, often ranking higher than raw linguistic features.

Key Results

The random forest models achieved impressive AUROC scores: 0.803 for English, 0.823 for French, and 0.829 for Spanish. These results represent a significant improvement over the baseline model (AUROC scores around 0.77 for English and French, and 0.74 for Spanish). The paper also provides a detailed breakdown of feature importances, with userMeanError and userVarError consistently ranking as the top predictors across all languages.

Limitations and Future Work

The authors acknowledge that the time-limited nature of the project constrained the extent of feature engineering and model optimization. They suggest several avenues for future work:

  • Ablation Studies and Model Comparisons: Further refining the feature set by identifying and removing less impactful features and exploring alternative modeling approaches like logistic regression or deep learning.
  • Investigating Interactions: Exploring more complex interactions between features, such as the interplay of user accuracy and token difficulty, or how prior language experience (potentially inferred from country data) influences learning.
  • Quantifying Benefits of Aggregated Features: Further analyzing the trade-offs between using aggregated error statistics versus dummy-encoded user/token IDs in terms of performance, time savings, and data requirements.
  • Expanding Feature Set: Incorporating additional features, such as word embeddings and more sophisticated syntactic structure analysis, to further enrich the model's predictive power.

Summary updated

BibTeX
@inproceedings{tomoschuk2018memory,
  author = {Brendan Tomoschuk and Jarrett Lovelett},
  title = {{A Memory-Sensitive Classification Model of Errors in Early Second Language Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure