Predicting Second Language Learner Successes and Mistakes by Means of Conjunctive Features

Yves BestgenNAACL 2018

TL;DR

This paper presents a system for predicting second language learner mistakes on individual words within exercises. By combining token-level features with metadata about the learner and the exercise (conjunctive features), the system achieved a respectable fifth place in the Duolingo SLAM challenge, demonstrating the value of these richer feature combinations.

What Problem Does This Paper Address?

The paper tackles the challenge of predicting learner errors in second language acquisition (SLA) at a granular, word-by-word level. Unlike traditional automated essay scoring (AES) which focuses on overall text quality, this task, known as Second Language Acquisition Modeling (SLAM), aims to pinpoint potential mistakes before they happen or as they occur within specific exercises. The Duolingo SLAM challenge provided a massive dataset of learner interactions, but with a key constraint: the actual learner answer was not provided, only the exercise and whether a mistake was made on a given token. This makes it distinct from AES and requires a different modeling approach.

Methods

The core innovation of this work lies in the use of conjunctive features. Instead of just looking at individual words or simple metadata, the system combines them. Specifically, it creates features by:

  • Token N-grams: Standard approach using sequences of words (unigrams, bigrams, trigrams) around the target token.
  • Metadata: Information about the learner (User ID), the exercise format (e.g., translation, fill-in-the-blank), and the session type.
  • Conjunctive Features: This is the novel part. They combine token n-grams with metadata (e.g., a specific trigram appearing for a particular user type) and also combine different metadata features themselves (e.g., User ID + Format + Session).

The models were trained using L1-regularized logistic regression (L1-LR) and later L2-LR for the final submission, chosen for their efficiency and performance. They also explored sequential information (e.g., how many times a token was seen, error rate on that token) but found it to be largely ineffective.

Datasets

The primary dataset used was from the 2018 Duolingo SLAM challenge. This dataset comprises a massive collection of learner exercises across three language pairs: English speakers learning Spanish (en_es), Spanish speakers learning English (es_en), and English speakers learning French (en_fr). The data was split into training (80%), development (10%), and testing (10%) sets. For model development and analysis, the authors focused on the smaller fr_en dataset due to faster processing times.

Key Insights and Ideas

  • Conjunctive Features are Crucial: The most significant finding is that simply looking at individual tokens or metadata isn't enough. Combining them (e.g., "this specific word sequence appearing in this type of exercise for this user") dramatically improves prediction accuracy.
  • Metadata is Necessary: The system significantly underperformed when only token n-grams were used, highlighting the importance of learner and exercise context.
  • Bigrams are More Useful than Trigrams: While trigrams were explored, bigrams proved to be more impactful for prediction.
  • Sequential Information Had Limited Impact: Surprisingly, features capturing the temporal aspect of learning (e.g., frequency of seeing a token, time since last seen) did not significantly improve performance. This suggests that static, contextual features were more powerful for this specific task.
  • Specific Error Patterns Emerge: The paper provides concrete examples showing how bigrams like "après_-" (after_) or "connaissez" (know) exhibit different error rates depending on the exercise type (reverse-tap vs. reverse-translate), illustrating the power of conjunctive features in capturing these nuances.

Key Results

The developed system ranked fifth out of 15 teams in the Duolingo SLAM challenge, achieving an average AUROC (Area Under the Receiver Operating Characteristic Curve) score. While not the top performer, it significantly outperformed the baseline and was competitive with other top teams. For the fr_en dataset, the base model achieved an AUROC of 0.8367. The analysis showed that conjunctive features, particularly those combining tokens with exercise format, made the most significant contribution to performance.

Limitations and Future Work

  • Lack of Longitudinal Modeling: The paper acknowledges that the model does not explicitly account for the longitudinal nature of the data (i.e., how learner competence evolves over time). A psycholinguistically motivated approach might yield better results here.
  • Ineffective Sequential Features: The authors found that their attempts to incorporate sequential information were inefficient and did not improve performance.
  • Limited Test Set Novelty: The test set contained a very small proportion of entirely new tokens not seen in training, which might have limited the evaluation of generalization capabilities.
  • Future Directions: The authors suggest exploring other metadata (like country of origin), incorporating L1 transfer knowledge, and potentially designing challenges with a higher proportion of unseen tokens to better test generalization. They also propose providing actual learner mistakes in training data for future research to develop models that can learn the token-mistake relationship more directly.

Summary updated

BibTeX
@inproceedings{bestgen2018predicting,
  author = {Yves Bestgen},
  title = {{Predicting Second Language Learner Successes and Mistakes by Means of Conjunctive Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure