JCT at SemEval-2021 Task 1: Context-aware Representation for Lexical Complexity Prediction

Chaya Liebeskind, Otniel Elkayam, Shmuel LiebeskindACL 2021

TL;DR

This paper presents a system for predicting lexical complexity of single words within their context, leveraging BERT's contextualized word embeddings alongside linguistic and statistical features. The authors found that while BERT embeddings are powerful, fine-tuning them for complexity prediction could further improve performance.

What Problem Does This Paper Address?

Predicting lexical complexity – how difficult a word is to understand – is crucial for applications like educational tools and text simplification. Traditionally, this was framed as a binary problem (complex/not complex). More recently, the task has evolved to predicting a probabilistic score or a Likert scale rating. The SemEval-2021 Task 1 on Lexical Complexity Prediction (LCP) aimed to address this by providing a new corpus (CompLex) with a 5-point scale and two sub-tasks: predicting complexity for single words and multi-word expressions. This paper focuses on the single-word prediction sub-task, aiming to build a system that accurately captures word complexity in context.

Methods

The authors employ a supervised Machine Learning (ML) approach. Their system integrates three main sets of features:

  • Linguistic Features: These include Part-of-Speech (POS) tags, dependency parsing relations, syllable counts, word length, frequency (from Google N-grams), and syntactic POS patterns. They also incorporate features derived from WordNet, such as the number of synsets for a word.
  • Statistical Features: Primarily focused on word and sentence length, and word frequency.
  • Semantic Features: This is where the novelty lies. The core of their semantic representation comes from BERT contextualized word embeddings. They average the BERT vectors of the target word and its surrounding context (three words before and after) to capture the semantic meaning. Additionally, they use K-Means clustering and k-Nearest Neighbors (KNN) on the unlabeled dataset to derive further features.

For feature selection, they experimented with several approaches, including removing features that decreased performance on the training set, iteratively removing highly correlated features, and Principal Component Analysis (PCA). They ultimately settled on a set of 101 features after filtering based on correlation with the complexity score.

Finally, they trained and evaluated five ML algorithms: Linear Regression, Support Vector Machine (SVM), XGBoost (XGB), KNN, and Stacking.

Datasets

The primary dataset used is CompLex, introduced for SemEval-2021 Task 1. This corpus is annotated using a 5-point Likert scale (1-5) and covers three genres: Bible translation, European Parliament proceedings, and biomedical articles. The authors used the train and trial sets for feature selection and model tuning, and the test set for final evaluation.

Key Insights and Ideas

  • BERT's Power: BERT's contextualized embeddings are a significant improvement over older, context-independent word embeddings for capturing semantic meaning relevant to lexical complexity.
  • Domain Specificity vs. Generalization: Surprisingly, using a domain-specific BERT model (BioBERT) for the biomedical domain did not outperform the general BERT model. This suggests that general language understanding from BERT is highly effective even across different domains for this task.
  • Feature Selection Matters: While BERT provides rich semantic features, careful feature selection (e.g., removing highly correlated features) was crucial for improving the performance of models like SVM and Linear Regression.
  • Ensemble Limitations: Stacking multiple classifiers did not yield better results than individual strong classifiers, suggesting that the chosen features might be largely capturing the same underlying information.

Key Results

The paper reports performance metrics like Pearson, Spearman, MSE, and R2. On the test set, the XGBoost algorithm achieved the best R2 score of 0.44, closely followed by SVM (0.439) and Stacking (0.433). Linear Regression and KNN performed slightly lower.

A confusion matrix analysis for XGBoost revealed that most classification errors occurred between "very easy" and "easy" words. The system also struggled with distinguishing between "neutral" and "easy" words. Notably, the "very difficult" class was not present in the test set, and the system did not predict any words as "very difficult."

Limitations and Future Work

The authors acknowledge that their system was not highly ranked in the competition. They suggest that fine-tuning BERT specifically for the complexity prediction task could significantly improve performance. They also believe that exploring other advanced feature selection approaches and potentially more sophisticated ensemble methods could be beneficial for future research.

Summary updated

BibTeX
@inproceedings{liebeskind2021jct,
  author = {Chaya Liebeskind and Otniel Elkayam and Shmuel Liebeskind},
  title = {{JCT at SemEval-2021 Task 1: Context-aware Representation for Lexical Complexity Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure