Sure, here's a blog-style summary of the paper:
TL;DR
This paper tackles the challenge of predicting lexical complexity in English text, a crucial step for applications like text simplification and language learning. The authors demonstrate that jointly training models on both single words and multi-word expressions (MWEs) improves performance, and that classic features like word frequency remain highly predictive even alongside modern contextual embeddings.
What Problem Does This Paper Address?
Understanding what makes a word or phrase "complex" for a reader is a fundamental problem in natural language processing. This complexity prediction is vital for making texts more accessible, whether for language learners, individuals with cognitive disabilities, or those with lower literacy skills. Previous work on Complex Word Identification (CWI) has often treated single words and multi-word expressions (MWEs) separately or used binary classifications. This paper addresses the gap by proposing a unified approach to predict lexical complexity on a continuous scale for both single words and MWEs, using a new dataset designed to overcome limitations of previous benchmarks.
Methods
The authors experimented with two types of regression models: a linear regression model and a multilayer perceptron (a type of neural network). They explored a rich feature set, combining:
- Contextualized Embeddings: Using BERT (Bidirectional Encoder Representations from Transformers) to generate context-aware representations for targets and their surrounding sentences. They found that averaging BERT embeddings for targets and contexts performed slightly better than concatenation.
- Hand-crafted Features: These included:
- BERT Vocabulary Token Count: A proxy for word frequency, as less frequent words are broken down into more sub-tokens by BERT's tokenizer.
- BERT Score for Masked Target: An indicator of how predictable a target is within its context.
- Morfessor Segmentation Length: The number of subwords a target is divided into, potentially reflecting morphological complexity.
- Subword Frequency: The average frequency of subwords within a target, aiming to capture morphological awareness.
- WordNet Synsets: A measure of word polysemy (how many different meanings a word has).
- Character Length: The length of the target in characters.
- Word Frequency: A classic and strong predictor of complexity.
They also investigated feature selection, reducing the feature set to the top 400 dimensions to improve model performance.
Datasets
The primary dataset used is CompLex, introduced in a previous work (Shardlow et al., 2020). This dataset is notable for:
- Providing complexity scores on a continuous scale (0 to 1).
- Containing both single words and MWEs.
- Annotations collected from native English speakers.
- Targets and contexts sourced from diverse domains (Bible, Europarl, biomedical texts).
The paper notes that CompLex was designed to address limitations of earlier CWI datasets like CWI 2016 and CWI 2018, which often used binary labels and had less defined MWE structures.
Key Insights and Ideas
- Joint Training is Beneficial: Training models on both single words and MWEs simultaneously leads to improved performance for both types of targets, especially for MWEs. This suggests that shared linguistic cues contribute to complexity across different lexical units.
- Classic Features Still Reign Supreme: Despite the power of contextual embeddings like BERT, traditional features like word frequency and BERT token count (a proxy for frequency) remain the most predictive features for lexical complexity. This highlights the enduring importance of statistical properties of words.
- MWEs Have Unique Complexity Drivers: While some features are universally important, others show differing importance. For instance, subword frequency is more predictive for single targets, while character length is more useful for MWE complexity. This suggests that the internal structure and length of MWEs play a more significant role in their perceived difficulty.
- Feature Importance Varies by Model Type: The paper hints that some features might be more suitable for linear models, while others benefit non-linear models, though this is not deeply explored.
Key Results
- Improved Performance with Joint Training: Tables 1 and 2 show that systems trained jointly on single words and MWEs generally achieve higher Pearson correlation coefficients (a measure of performance) compared to systems trained separately. For example, the joint linear model achieved 0.785 for MWEs, while the separate linear model achieved 0.678.
- Non-linear Models Slightly Outperform: The neural network (NN) models consistently showed a slight advantage over linear regression models, particularly for MWEs.
- Feature Selection Boosts Linear Models: Reducing the feature set to the top 400 dimensions improved the performance of the linear model.
- Word Frequency is King: Word frequency was consistently ranked as the most important feature across various setups.
Limitations and Future Work
- Hyperparameter Tuning: The authors acknowledge that hyperparameters for the neural network were not extensively tuned, suggesting that further optimization could yield better results.
- Subword Frequency Underperformance: The subword frequency feature did not perform as well as expected, potentially due to the limited amount of data used for its estimation. The authors propose collecting statistics from a larger resource to amend this.
- Understanding Joint Training Success: Further research is needed to fully understand why joint training with single words and MWEs is successful. Is it due to the smaller MWE subcorpus size, or the inherent nature of MWEs?
- MWE Structure: Investigating the specific contribution of the second word in noun-noun or adjective-noun MWEs to overall complexity is a promising avenue.
Summary updated