TL;DR
This paper presents a simple yet effective system for predicting lexical complexity, achieving strong results in the SemEval-2021 Task 1. The authors demonstrate that a carefully engineered set of linguistic features, rather than complex models or extensive hyperparameter tuning, is sufficient for high performance, highlighting the importance of feature engineering.
What Problem Does This Paper Address?
Lexical complexity, the inherent difficulty of understanding a word, is crucial for various applications like educational material design, text simplification, and reading comprehension analysis. While previous work often focused on identifying "complex" words as a binary classification (Complex Word Identification or CWI), the SemEval-2021 Lexical Complexity Prediction (LCP) Shared Task introduced a new dataset (CompLex) that allows for continuous complexity prediction. This shift requires models to predict a graded complexity score rather than a simple yes/no. The challenge lies in identifying the most informative features and building a system that balances performance with simplicity, avoiding overly complex models that might be difficult to interpret or deploy.
Methods
The authors focused heavily on feature engineering rather than complex model tuning. They explored three main categories of features:
- Word and Sentence-Level Linguistic Features: This includes basic linguistic properties like character, syllable, and phoneme counts, part-of-speech tags, named entity tags, and language learning-related metrics such as age of acquisition and word prevalence. Sentence-level features included lexical diversity, average sentence length, and readability scores like Linsear Write.
- Frequency Features: They utilized log relative frequency, raw frequency, frequency rank, and ZIPF frequency from various corpora.
- Word Embeddings: Pre-trained embeddings like fastText and BERT were also investigated.
For modeling, they experimented with several regression algorithms, including linear regression, Multi-Layer Perceptron, Bagging, Stacking, Random Subspace, M5P, and Random Forest. Their best-performing model was trained using the Random Forest classifier with default hyperparameters. They also performed extensive ablation studies to determine the most impactful features and feature combinations.
Datasets
- CompLex Corpus: This is the primary dataset used for the LCP Shared Task 2021. It provides sentences with target words and their complexity scores on a 5-point Likert scale, annotated by multiple native English speakers. The dataset consists of 7,662 training and 917 testing instances.
- Additional Corpora for Feature Extraction: The authors used nine corpora for word frequency extraction, including BNC, Simple Wikipedia, English Wikipedia, SubIMDB, and various corpora from the OPUS project (bible-uedin, EMEA, Europarl, News-Commentary, OpenSubtitles 2018).
- Word Lists: They also incorporated two word lists with annotated CEFR levels and the Age of Acquisition dataset.
Key Insights and Ideas
- Simplicity Wins: The core message is that a well-engineered feature set can outperform complex models. Their simplest, yet highly effective, model used only 36 linguistic features.
- Linguistic Features are Key: While word embeddings were explored, they found that linguistic features were more informative and less computationally expensive.
- Frequency Matters Most: Frequency-based features, particularly from daily spoken language sources like film subtitles, consistently proved to be highly informative.
- Contextual Information is Helpful, but Marginally: Including sentence-level context improved performance, but the gains were relatively small, suggesting that a single sentence might not capture all necessary contextual nuances for complexity prediction.
- Feature Engineering Over Model Tuning: The authors explicitly state their focus was on feature engineering, and they achieved excellent results using default classifier settings.
Key Results
- The best system, trained on 112 features (including word embeddings), achieved a Pearson's score of 0.7942.
- A simpler system, trained on just 36 linguistic features, achieved a comparable Pearson's score of 0.7925. This simpler system was submitted to the task.
- The submitted system achieved a Pearson's score of 0.7588 on the official test set, ranking in the upper half of the submitted systems.
- The baseline system (linear regression with log relative frequency and word length) achieved a Mean Absolute Error (MAE) of 0.0867. The authors' M5P model achieved an MAE of 0.0638 and a Pearson's score of 0.7811, outperforming the baseline.
Limitations and Future Work
- Limited Context: The current system primarily uses single sentences for context. The authors suggest that future research could explore incorporating more extensive contextual information.
- Reader Perception: The paper acknowledges that word complexity is subjective and can differ between readers. Future work could target specific reader groups (e.g., people with dyslexia, second language learners) and incorporate their background information into annotation and experimentation.
- Computational Cost of Embeddings: While word embeddings were explored, their computational cost and relatively lower informativeness compared to linguistic features led the authors to favor simpler models. Future work might explore more efficient ways to leverage embeddings or focus on their specific strengths.
Summary updated