TL;DR
This paper introduces SemEval-2021 Task 1, a new challenge focused on predicting the complexity of individual words and multi-word expressions on a continuous scale, moving beyond binary classification. The task utilizes the CompLex 2.0 dataset, featuring contextualized annotations across diverse genres, and attracted 55 participating teams, showcasing advancements in lexical complexity prediction.
What Problem Does This Paper Address?
Comprehending text is significantly impacted by the presence of unknown or difficult words. Traditional approaches to identifying these "complex" words, like Complex Word Identification (CWI), often rely on binary (complex/not complex) or probabilistic judgments. However, this binary approach can be overly simplistic and doesn't capture the nuanced degree of difficulty a word might present. Readers might struggle with a word due to unfamiliarity, even if they can infer its meaning, or find it completely incomprehensible. This paper addresses this limitation by proposing and evaluating a Lexical Complexity Prediction (LCP) task, which aims to assign a continuous score (0-1) to words, reflecting their complexity more accurately. This finer-grained prediction is crucial for applications like lexical simplification and readability assessment, where understanding the degree of difficulty is as important as knowing if a word is difficult.
Methods
The core of this work is the introduction of the Lexical Complexity Prediction (LCP) task, which differs from prior CWI tasks by using a continuous Likert scale for annotation. Annotators rated words on a 5-point scale (Very Easy to Very Difficult), which was then converted to a continuous scale (0-1). This approach aims to capture more nuanced judgments than binary classification. The paper also details the creation of the CompLex 2.0 dataset, an augmented version of the CompLex corpus, which includes contextualized annotations across three genres (Europarl, Biomedical, and Bible). Participants in the SemEval task were tasked with building systems to predict these continuous complexity scores. The evaluation metrics used for ranking were primarily Pearson's correlation, with Spearman's rank, Mean Absolute Error (MAE), Mean Squared Error (MSE), and R2 also considered.
Datasets
The primary dataset used is CompLex 2.0. This dataset is an augmented version of the previously published CompLex dataset. It contains approximately 10,800 instances, spanning three genres: Europarl (parliamentary proceedings), Biomedical literature, and the Bible. The dataset includes both single words (unigrams) and multi-word expressions (MWEs, modeled as bigrams). Annotations were collected using crowdsourcing on Amazon's Mechanical Turk, with each instance receiving multiple annotations, and the average score serving as the gold label. The dataset was split into trial, training, and test sets with careful stratification to ensure representativeness across genres, complexity levels, and tokens.
Key Insights and Ideas
- Continuous Complexity is More Informative: Moving from binary to continuous complexity scores provides a richer representation of word difficulty, enabling more sophisticated downstream applications.
- Context is Crucial: The inclusion of sentence context in the dataset and the success of systems that utilized context highlight its importance for accurately predicting lexical complexity.
- MWEs are More Complex: Multi-word expressions were found to be significantly more complex on average than single words, indicating a greater challenge for readers.
- Deep Learning Dominates, but Feature-Based Methods Remain Relevant: While deep learning approaches, particularly those leveraging pre-trained language models like BERT and RoBERTa, achieved the highest scores, feature-based systems still performed competitively, suggesting a hybrid approach might be optimal.
- Annotation Subjectivity is Inherent: Despite efforts to standardize, annotator subjectivity remains a factor. The paper acknowledges that averaging judgments helps, but further work could explore how to model this subjectivity.
Key Results
The SemEval-2021 Task 1 attracted 55 teams, with 54 submitting results for Sub-task 1 (single words) and 37 for Sub-task 2 (single words and MWEs). The winning systems demonstrated strong performance, with the top system achieving a Pearson's correlation of 0.7886 for Sub-task 1 and 0.8612 for Sub-task 2. High R2 scores were also reported, indicating that the models effectively captured the variance in the data. The results showed that systems generally performed better on Sub-task 2, likely due to the inherent complexity difference between single words and MWEs. The performance of systems significantly outperformed a baseline based on word frequency.
Limitations and Future Work
- Dataset Representativeness: The CompLex 2.0 dataset was pre-selected for harder words, meaning it is not representative of the general distribution of words in the English language. Future work could involve annotating all words without filtering to create a more representative corpus.
- Subjectivity in Annotations: While averaging annotations mitigates subjectivity, individual differences in perception of complexity persist. Future research could explore how to model this subjectivity.
- Language Specificity: The task and dataset are currently focused solely on English. Extending this task to other languages is a promising direction for future work.
- Evaluation Methodology: The paper notes that small variations in dataset splits can lead to different rankings among top-performing systems, suggesting a need for community-wide discussion on robust evaluation methodologies for NLP tasks.
Summary updated