TL;DR
This paper introduces two convolutional neural networks (CNNs) for identifying complex words and phrases in context, achieving a Pearson correlation of 0.7754. The models leverage word and character embeddings alongside lexical features, demonstrating a promising approach to a task crucial for text simplification and accessibility.
What Problem Does This Paper Address?
Complex Word Identification (CWI) is the task of pinpointing words or phrases that a reader might find difficult to understand. The complexity of a word isn't static; it depends heavily on context and can even involve general knowledge for named entities. This is a significant challenge because accurately identifying complex words is vital for applications like simplifying text for individuals with conditions such as dyslexia, thereby improving literature accessibility. Previous CWI tasks often treated complexity as a binary (complex/not complex) or probabilistic problem, but human judgment is more nuanced, existing on a continuous scale. The SemEval-2021 Task 1, Lexical Complexity Prediction (LCP), aimed to address this by using an augmented dataset with a 5-point Likert scale and context-specific annotations across different domains (Bible, Biomed, Europarl).
Methods
The authors developed two CNN models: one for single words and another for multi-word expressions (MWEs). Both models utilize a combination of features:
- Word and Character Embeddings: Pre-trained 50d GloVe embeddings for words and 50d chars2vec embeddings for character sequences are used. The latter helps infer relationships between words with similar morphologies.
- Lexical Features: These include word frequency (from SUBTLEX-UK), age of acquisition (AoA), word length, and number of syllables.
- Corpus Type: A one-hot encoding representing the domain (Bible, Biomed, Europarl) is also included.
For preprocessing, features are min-max normalized, and non-alphanumeric characters are removed. Both models take the target word's features and averaged features from its left and right contexts as input. For MWEs, the models are adapted to handle two-word phrases. The single-word model uses a 1D convolutional layer followed by fully connected layers. The multi-word model builds upon the single-word model, acting as an adapter that feeds into a pre-trained single-word model.
Datasets
The primary dataset used is CompLex, an augmented version of a previous dataset, annotated with a 5-point Likert scale for word complexity and featuring context-specific annotations. The dataset spans three domains: Bible, Biomed, and Europarl. The models were trained and evaluated on this dataset for SemEval-2021 Task 1.
Key Insights and Ideas
- Context is King, but Averaging is a Simplification: While CNNs can leverage contextual information, averaging left and right context features means all words in the context are weighted equally. This can be a limitation, as some words might be more influential than others, and irrelevant stop words can dilute the signal.
- Out-of-Vocabulary (OOV) Words as a Signal: The model surprisingly showed a better correlation for OOV words, suggesting that the lack of a word embedding itself acts as a useful feature for predicting complexity. This might be because OOV words are often more complex, but it could also lead to false positives for low-complexity OOV words.
- MWEs are Harder, Especially with Less Variance: The multi-word model struggled more with the Europarl corpus, which had a smaller standard deviation in complexity values compared to the other domains. This suggests that models might find it harder to adapt when the range of complexity is narrower.
- Adjective-Noun MWEs are Easier: The model performed marginally better on Adjective-Noun (A-N) MWE formations compared to Noun-Noun (N-N) formations.
Key Results
- The overall system achieved a Pearson correlation of 0.7754 and an R² of 0.5989 on Task 2 (single and two-word MWEs).
- The single-word model achieved a Pearson correlation of 0.7389 and an R² of 0.5398 on Task 1 (individual words).
- Performance varied significantly across different domains, with the JUST-BLUE corpus (likely a combination or specific subset) showing a high Pearson correlation of 0.7886 for individual words.
- The multi-word model showed a Pearson correlation of 0.7611 across all domains.
- The model struggled with predicting complexity for very simple words and MWEs (complexity < 0.1), likely due to fewer examples in the training data.
Limitations and Future Work
The authors acknowledge that averaging context features limits the model's ability to discern the true relevance of individual words within the context. They suggest that incorporating mechanisms to weight context words based on specific features could improve performance. Furthermore, they propose that attention-based models like BERT, which excel at capturing rich contextual information, could be fine-tuned for CWI tasks to produce more effective embeddings. The current multi-word model is limited to two-word MWEs, and adapting it to handle longer sequences would be a valuable extension.
Summary updated