TL;DR
This paper tackles the challenge of predicting how complex a word or phrase is within a given sentence. The authors propose combining statistical features, character-level language models, and masked language models (like BERT) to achieve better predictions, demonstrating that these diverse approaches can be effectively merged for improved lexical complexity assessment.
What Problem Does This Paper Address?
Understanding the complexity of language is crucial for various applications, from educational tools to accessibility features. SemEval-2021 Task 1 specifically focused on predicting the "lexical complexity" of target tokens (words or multi-word expressions) within English sentences. This complexity is rated on a scale from 0 (not very complex) to 1 (very complex). For instance, "land" in "Our land will yield its increase" is considered simple (0.19), while "doxycycline" in "The reason these two lines were unresponsive to doxycycline is unknown" is more complex (0.75). The task is divided into two sub-tasks: predicting complexity for single words (SINGLE) and for multi-word expressions (MULTI). The challenge lies in capturing the nuances of word complexity, which is influenced by factors like word frequency, its role in the sentence, and its inherent structure.
Methods
The researchers explored three main approaches to predict lexical complexity:
- Statistical Baseline Features: This involved extracting seven features, including the length of the target token, its frequency in English word frequency databases (SubtlexUS and SubtlexUK), whether it's a multi-word expression (MWE), and indicators for the source corpus (bible, parliamentary proceedings, biomedical text). For MWEs, frequencies were averaged across component words.
- Character-Level Encoder: This method used a bidirectional GRU (Gated Recurrent Unit) language model. The first GRU processed the entire sentence to create a sentence embedding, and a second GRU processed the target token, using the sentence embedding to initialize its hidden state. The final hidden state of this second GRU was used as a representation of the target word in context. The hypothesis here is that the model can learn the target word's complexity by considering its sentence context.
- Masked Language Model (BERT): Leveraging the power of BERT (Bidirectional Encoder Representations from Transformers), the authors replaced the target token in a sentence with the special
[MASK]token. The probability of the original target token being predicted by BERT in this masked context was then used as a feature. The intuition is that language models like BERT, trained on vast amounts of text, implicitly learn about word difficulty and predictability. For multi-word expressions, the probabilities of their constituent tokens were averaged.
These individual approaches were then combined in various ways to form their submitted systems.
Datasets
The primary dataset used for this task was introduced in Shardlow et al. (2020) and consists of sentences from three sources:
- A translated bible
- European Parliament proceedings
- A biomedical corpus
This dataset was used for both training and evaluation, with the paper detailing experiments on a validation set and referring to results on a test set for the official shared task ranking.
Key Insights and Ideas
- Synergy of Diverse Features: The most significant insight is that combining information from different sources—statistical, character-level, and masked language models—yields superior results compared to using any single approach alone. This highlights that lexical complexity is a multifaceted phenomenon that requires a holistic view.
- BERT's Implicit Knowledge: The success of the BERT-based approach suggests that large language models, even when used in a relatively straightforward manner (predicting masked token probability), capture a substantial amount of information relevant to lexical complexity.
- Character-Level Context Matters: The character-level encoder's contribution, especially when combined with other methods, indicates that the internal structure of words and their immediate sentence context are vital for assessing complexity.
- Task-Specific Performance Variations: The paper notes interesting variations in performance across evaluation metrics for the SINGLE task, suggesting that different metrics might capture different aspects of complexity, and that models might excel in some areas while struggling in others.
Key Results
The authors submitted three systems to SemEval-2021 Task 1. Their best-performing system (System 2) combined the character-level encoder and the masked language model approach, feeding this combined representation into a Support Vector Regressor (SVR).
- Overall Best Performance: System 2 achieved the best results on the MULTI sub-task across all evaluation metrics, including Pearson correlation (R), Spearman correlation (Rho), Mean Absolute Error (MAE), Mean Squared Error (MSE), and R-squared (R2).
- SINGLE Sub-task Performance: For the SINGLE sub-task, System 2 also performed best in terms of Pearson correlation (R), which was the primary metric for ranking in the shared task. However, System 3 (which combined baseline features, character-level encoder, and masked language model into a feed-forward network) outperformed System 2 on MAE, MSE, and R2 for the SINGLE task, indicating a trade-off in performance depending on the metric.
- Improvement over Baselines: All submitted systems consistently outperformed a simple logistic regression baseline trained on just the statistical features.
Limitations and Future Work
- Metric Sensitivity: The authors acknowledge that their best model (System 2) showed substantial variation in performance across different evaluation metrics on the SINGLE sub-task. This suggests that future work could further explore the model's performance variations and potentially develop models that are more robust across various metrics.
- Relationship Between Sub-tasks: The paper suggests that future work should investigate the relationship between the SINGLE and MULTI sub-tasks, as well as how different models and evaluation metrics interact with the type of target expression (single vs. multi-word).
- Exploration of Target Expression Type: Further research could delve into how the specific characteristics of the target expression (e.g., its semantic category, part of speech) influence model performance.
Summary updated