TL;DR
This paper presents an ensemble approach combining feature-based and neural models for predicting lexical complexity (how difficult a word is to understand). Their best system achieved a Pearson correlation of 0.7556 on the SemEval-2021 test set, demonstrating the power of combining diverse modeling techniques for this task.
What Problem Does This Paper Address?
The paper tackles the problem of Lexical Complexity Prediction (LCP), which aims to identify words that are difficult for a target population to understand. This has practical applications in areas like creating texts suitable for language learners or simplifying content for individuals with low literacy. Previous work often framed this as a binary classification (complex/not complex), but this paper, following recent trends, treats it as a regression problem, predicting a continuous complexity score. The SemEval-2021 shared task provided a new multi-domain dataset annotated with a 5-point Likert scale, moving beyond binary judgments to a more nuanced understanding of complexity.
Methods
The authors developed four distinct models that were later ensembled:
- Random Forest Regressor: A baseline model using 15 hand-crafted linguistic features. These features were inspired by psycholinguistic studies and readability metrics, including word length (syllables, characters), frequency (from Simple English Wikipedia), and syntactic properties (dependency labels, POS tags).
- Fine-tuned BERT: A standard approach where a pre-trained BERT model is fine-tuned for the regression task. The input consists of the target word and its surrounding sentence.
- Frequency Pre-trained BERT: An extension of the fine-tuned BERT model. Before fine-tuning on the LCP task, the BERT model is further pre-trained on a custom dataset derived from Simple English Wikipedia, specifically focusing on word frequency and part-of-speech combinations. This leverages the Zipfian distribution of word frequencies.
- Neural Multi-Task Learning (MTL): This model uses a DistilBERT base and incorporates auxiliary tasks alongside the primary LCP regression objective. The auxiliary tasks included Part-of-Speech (POS) tagging, Grammatical Relations (GR) prediction, and genre classification. The idea is that learning these related tasks can improve the model's understanding of language and, consequently, its LCP prediction performance.
The final submission was an ensemble of these four models, combining their predictions.
Datasets
- CompLex dataset: The primary dataset used for training and evaluation. This is an augmented version of a previously existing dataset, featuring sentences annotated with a 5-point Likert scale for lexical complexity. The complexity labels were normalized to the [0, 1] range. The dataset comprises texts from three genres: Bible, Biomedical, and Europarl.
- Simple English Wikipedia: Used for feature engineering (frequency counts) and for creating a custom pre-training dataset for one of the BERT models.
Key Insights and Ideas
- Ensembling is Key: The paper strongly advocates for ensembling diverse models. Their best performing system was an ensemble of all four individual models, highlighting that combining different approaches (feature-based and neural, with various neural architectures) yields superior results.
- Neural Models Outperform Traditional Features: Contrary to some previous findings in similar tasks, the neural models (BERT, BERT-freq, MTL) significantly outperformed the feature-based Random Forest model, suggesting a shift in the state-of-the-art.
- Frequency Matters (but context is crucial): While frequency features were important for the Random Forest model, the success of the neural models, especially the BERT-freq variant, underscores the importance of contextual information and pre-trained language representations.
- MTL Shows Promise: The Multi-Task Learning approach demonstrated strong performance, suggesting that auxiliary tasks can indeed help the model learn more robust representations for lexical complexity.
- Genre Differences Exist: Performance varied across genres, with the Biomedical data showing a slightly larger interquartile range in complexity scores, potentially impacting metrics like MAE.
Key Results
- Overall Performance: The final ensemble system achieved a Pearson correlation of 0.7556 on the test set.
- Individual System Performance (Trial Set):
- BERT freq. achieved the best Pearson score (0.8089).
- BERT achieved the best Spearman score (0.7579).
- The ensemble of all systems achieved the best Pearson (0.8264) and Spearman (0.7676) on the trial set.
- Ensemble Performance (Trial Set): The best ensemble (MTLAI + BERT + BERTfreq.
- RandomForest) achieved a Pearson score of 0.8264.
- Test Set Performance: The final ensemble achieved a Pearson score of 0.7556.
- Ablation Study: Removing the Random Forest model from the ensemble improved the test set performance (Pearson 0.7560 vs. 0.7556), suggesting it was detrimental on the test set, though it contributed positively on the trial set. This highlights potential differences between the trial and test data distributions.
Limitations and Future Work
- Trial vs. Test Set Discrepancy: The authors noted an inconsistency in the performance of the Random Forest model between the trial and test sets. They hypothesize this is due to the smaller sample size of the test set and noise in the data, making the weak correlations of features less reliable.
- Feature Importance: While frequency of lemma and word were important for the Random Forest, their overall contribution was limited compared to neural models.
- Future Work: The paper doesn't explicitly detail future work, but the success of ensembling and MTL suggests these directions are promising. Further investigation into the trial/test set discrepancies and exploring more sophisticated feature engineering for neural models could be avenues for future research.
Summary updated