TL;DR
This paper tackles the challenge of predicting lexical complexity for single words, a crucial step in making text more accessible. The authors propose a novel "stacking" approach that combines predictions from multiple fine-tuned Transformer models with traditional linguistic features, achieving near-state-of-the-art results. Their findings highlight the power of combining diverse models and the surprising benefit of incorporating Word Sense Disambiguation data for this task.
What Problem Does This Paper Address?
Making text understandable for a wider audience often involves simplifying complex words. To automate this process, researchers need to accurately identify which words are "complex." This paper focuses on Lexical Complexity Prediction (LCP), specifically for individual words within their sentence context. Unlike previous work that often treated this as a binary "complex/not complex" problem, the SemEval-2021 Task 1 (Sub-task 1) framed it as predicting a continuous complexity score. A key challenge is that a word's complexity can vary significantly depending on the surrounding sentence, meaning context is paramount.
Methods
The core of the IAPUCP system is a stacking ensemble approach. They start by leveraging several pre-trained Transformer models (RoBERTa and XLNet) that are fine-tuned on the LCP task data. Crucially, they experiment with different context window sizes (1 or 2 tokens around the target word) for these fine-tuned models.
A novel aspect of their approach is the use of multi-task learning for one of the BERT models. This model is trained simultaneously on both the LCP task and Word Sense Disambiguation (WSD). The hypothesis is that understanding a word's different senses can inform its complexity.
These Transformer-based predictions are then combined with a set of traditional morphosyntactic and lexical features. These include character and word counts, part-of-speech tags, syntactic dependencies, n-gram frequencies from various corpora (Children's Book Test, Simple Wikipedia, Lang-8, Tatoeba), and a pre-existing word complexity lexicon.
Finally, all these features (Transformer predictions and traditional features) are fed into a LightGBM gradient boosting model, which acts as the final "stacker" to produce the complexity score.
Datasets
The primary dataset used is from the SemEval-2021 Task 1: Lexical Complexity Prediction. This dataset is an extension of the CompLex dataset, which was built using Likert scale ratings for word complexity. It includes data from three domains: Bible, Europarl, and Biomed. The task was split into two sub-tasks: single words (which this paper addresses) and multi-word expressions. The authors also utilized data for Word Sense Disambiguation (WSD) for their multi-task learning approach.
Key Insights and Ideas
- Stacking is Powerful: Combining predictions from multiple Transformer models and traditional features significantly outperforms any single model. This emphasizes the benefit of diverse feature sets and model architectures.
- Multi-task Learning for WSD Boosts LCP: Training a BERT model on both LCP and WSD tasks yielded the most impactful feature for the final LightGBM model. This suggests that understanding word senses is a valuable signal for predicting lexical complexity.
- Context Matters (and Window Size is Key): Fine-tuning Transformers with different context window sizes (1 or 2 tokens) provided distinct and valuable signals, both of which contributed to the final model's performance.
- Biomed Corpus is Tricky: While their model performed best on the Biomed corpus in validation, it also showed higher variance. This suggests that predicting complexity in domains with potentially more nuanced or varied language can be more challenging.
Key Results
The IAPUCP system achieved a Pearson correlation of 0.7704 on the test set of the SemEval-2021 Task 1 (Sub-task 1). This placed them 10th overall, only 0.018 points behind the winning submission. An ablation study confirmed that all individual components (Transformer predictions and traditional features) contributed to the final performance, with the multi-task BERT model showing the greatest impact.
Limitations and Future Work
The authors acknowledge that the task, especially with unseen tokens, resembles a zero-shot classification problem. They suggest that semi-supervised learning or data augmentation techniques could be explored as future work to improve performance. Additionally, they propose extending their multi-task learning approach to other Transformer-based models.
Summary updated