TL;DR
This paper presents the winning approach for SemEval-2021 Task 1, which focuses on predicting the lexical complexity of words within sentences. The team, JUST-BLUE, achieved top performance by leveraging pre-trained language models BERT and RoBERTa and employing a weighted averaging ensemble method. This work highlights the effectiveness of advanced deep learning models for nuanced linguistic analysis.
What Problem Does This Paper Address?
Predicting the complexity of words is a crucial yet challenging task in Natural Language Processing (NLP). Understanding lexical complexity is vital for applications like text simplification, rearrangement, and improving overall text readability and comprehension. Previous research has approached this problem using binary classification (complex vs. not complex) or regression models. However, SemEval-2021 Task 1 aimed to predict lexical complexity on a continuous scale (0-1), requiring a more granular understanding of word difficulty within context. This task builds upon existing datasets like CompLex, but with an augmented version and a focus on predicting complexity within sentences.
Methods
The JUST-BLUE team's approach centers on utilizing powerful pre-trained language models, specifically BERT and RoBERTa. They recognized that a word's complexity is influenced not only by the word itself but also by the sentence it appears in. Therefore, their strategy involved training two sets of models:
- Token-level models: These models were trained to predict the complexity of individual words (tokens).
- Sentence-level models: These models were trained to predict the complexity of the entire sentence containing the word.
The outputs from these BERT and RoBERTa models were then combined using an ensemble method. Initially, they explored simple averaging, but found that a weighted averaging approach yielded better results. This method allowed them to assign different importance levels to the token-level and sentence-level predictions. Specifically, they found that an 80% weight for token-level predictions and 20% for sentence-level predictions provided the optimal balance. This strategy acknowledges that while the word itself is primary, its surrounding context significantly modulates its perceived complexity.
Datasets
The primary dataset used for this task was the augmented version of CompLex, provided by SemEval-2021 Task 1. This dataset consists of sentences annotated with a 5-point Likert scale (1-5) indicating word complexity, which was then mapped to a 0-1 scale for the task. The data was sourced from three different corpora: the Bible, biomedical texts, and the European Parliament. The dataset was split into trial, train, and test sets.
Key Insights and Ideas
- Context is King (but the word is the kingdom): The most significant insight is that while the intrinsic complexity of a word is important, its surrounding sentence context plays a crucial role in determining its perceived difficulty. The weighted averaging ensemble effectively captured this by giving more weight to token-level predictions while still incorporating sentence-level information.
- Pre-trained models are powerful: BERT and RoBERTa, as state-of-the-art pre-trained language models, proved highly effective for this nuanced NLP task, significantly outperforming traditional machine learning models like SVM and Random Forest.
- Ensembling is essential: Combining the predictions of multiple models, especially those trained on different aspects of the data (token vs. sentence), is a robust strategy for improving performance and achieving higher accuracy.
- Hyperparameter tuning matters: While the core models were powerful, fine-tuning hyperparameters like learning rate, batch size, and sequence length was critical for optimizing performance.
Key Results
The JUST-BLUE team achieved the first rank in the SemEval-2021 Task 1 (LCP-subtask1) for predicting lexical complexity. Their model, JUST-BLUE, obtained a Pearson correlation score of 0.788, which was the highest among all participants. They also reported a Mean Absolute Error (MAE) of 0.0609, indicating high accuracy in their predictions. The Spearman's Rho (Rho) and R-squared (R2) scores were 0.7369 and 0.6172, respectively. Their ensemble approach, weighting token-level predictions at 80% and sentence-level at 20%, proved to be the most effective configuration.
Limitations and Future Work
The paper notes that the "large size" models of BERT and RoBERTa sometimes decreased accuracy, suggesting that for this specific task, smaller or fine-tuned versions might be more suitable. The authors also mention that the test set contained new words, which necessitated limiting arguments to avoid overfitting. Future work could explore more sophisticated ensembling techniques, investigate other pre-trained models, or delve deeper into analyzing the specific linguistic features that contribute most to lexical complexity. Further research could also focus on adapting these models to different languages or domains.
Summary updated