TL;DR
This paper tackles the challenge of automatically predicting the lexical complexity of Swedish words for second language learners. By leveraging existing graded vocabulary lists and incorporating topic information, the authors achieve improved accuracy in classifying words by CEFR level, paving the way for more personalized language learning materials.
What Problem Does This Paper Address?
Developing effective digital learning materials for second language acquisition, especially for the growing number of immigrants to Sweden, requires a robust understanding of vocabulary complexity. Ideally, learning materials should be structured to gradually introduce new words based on a learner's proficiency. While graded vocabulary lists exist, they are inherently finite. This paper addresses the gap by proposing a method to predict the CEFR (Common European Framework of Reference for Languages) level of unseen words, thereby extending the utility of existing graded lists and enabling the automatic generation of learning exercises tailored to specific proficiency levels.
Methods
The core of this work involves training machine learning models to predict a word's CEFR level. The authors extract a rich set of features from words, categorizing them into:
- Count-based and surface form features: These include basic word properties like length, syllable count, and the presence of non-alphanumeric characters or numbers. They also incorporate features related to multi-word expressions and character-level bigrams.
- Morphological features: This category includes the part-of-speech of a word, suffix length (hypothesizing that knowledge of derivational morphology is acquired later), and compound count (the number of possible interpretations of a word through compounding).
- Semantic features: Here, they consider the degree of polysemy (how many different meanings a word has) and homonymy (how many words share the same spelling but have different meanings).
- Context features: Crucially, they introduce topic distributions, indicating which topics a word is associated with based on its occurrence in a corpus.
They experiment with several classifiers, including Support Vector Machines (SVM), Multi-Layer Perceptrons (MLP), and Extra Trees (ET), and evaluate their performance using 10-fold cross-validation. A key innovation is the inclusion of topic distributions as features, which they hypothesize will improve classification accuracy.
Datasets
The study utilizes three Swedish word lists:
- SVALex: A receptive vocabulary resource compiled from the COCTAILL textbook corpus, graded for CEFR levels.
- SweLLex: A productive vocabulary resource derived from the pilot SweLL learner essay corpus, also graded for CEFR levels.
- Kelly list: A resource containing the 8425 most frequent lemmas from native speaker web writing, graded by CEFR level based on frequency and corpus coverage.
The paper notes that SVALex and SweLLex have overlapping vocabulary items across levels, and Kelly list covers six proficiency levels (A1-C2), with C2 being assimilated to C1 for consistency.
Key Insights and Ideas
- Topic information is crucial: Adding topic distributions as features significantly improves classification accuracy, suggesting that the context in which a word appears is a strong indicator of its complexity for learners.
- Feature interactions matter: While some individual features like lexical frequency (implicitly captured by n-gram probabilities) are known predictors, the combination of various features (morphological, semantic, and contextual) provides a more comprehensive picture of word complexity.
- Polysemy and homonymy are complex indicators: The paper highlights that while non-polysemous words are often less frequent, highly polysemous words are learned early. This suggests a nuanced relationship between semantic features and acquisition order.
- Data sparsity is a challenge: The authors acknowledge that the limited size of some corpora (e.g., COCTAILL) can lead to sparser data, potentially affecting the reliability of predictions.
Key Results
- The Extra Trees classifier with topic distributions added achieved the highest accuracy of 0.55 ± 0.05 on SVALex (receptive vocabulary) in a five-way classification.
- Without topic distributions, the best system achieved 0.32 accuracy on SVALex, which is only slightly better than the majority baseline (0.29).
- When comparing their results to previous work on French (Gala et al., 2014), their system achieved 45% accuracy on L1 data (Kelly list) compared to 63% in the previous study, and 59% accuracy on L2 data (SweLLex) compared to 43% in the previous study. They attribute this difference to the size and nature of the corpora used.
- The paper also presents a user interface that allows users to input a word and receive a predicted CEFR level, demonstrating the practical application of their model.
Limitations and Future Work
- Data Sparsity: The authors acknowledge that the limited size of some corpora can lead to sparser data and affect prediction reliability. They plan to address this by collecting and annotating more data.
- Sense-level complexity: The current models assign a single level to a word, regardless of its different meanings. Future work will involve recalculating resources at the sense level by incorporating word sense disambiguation.
- Missing features: Features like the number of phonemes are not currently included but could be valuable given Swedish's non-transparent grapheme-to-phoneme correspondences.
- Contradictory data points: The vector representation of words can lead to different words having the same representation, creating contradictory data points. Future work may involve adding more disambiguating features.
- Validation: The authors plan to evaluate their models with teachers and language learners to assess the validity of the automatic mapping and create gold-standard annotations.
- Crowdsourcing: They propose embedding words in automatically generated learner exercises to implicitly crowdsource learner knowledge and refine predictions.
Summary updated