RS_GV at SemEval-2021 Task 1: Sense Relative Lexical Complexity Prediction

Regina Stodden, Gayatri VenugopalACL 2021

TL;DR

This paper introduces RS_GV, a neural network system for predicting the complexity of English words on a scale of 0 to 1. By combining hand-crafted linguistic features, contextualized word and character embeddings, and a novel sense-relative normalization approach, RS_GV achieves competitive results on the SemEval-2021 Task 1, particularly excelling with biomedical terms.

What Problem Does This Paper Address?

The paper tackles the challenge of Lexical Complexity Prediction (LCP), a crucial sub-task in text simplification. Text simplification aims to make written content more accessible by replacing complex words with simpler synonyms. While previous approaches focused on binary classification (complex vs. simple), LCP aims for a more nuanced understanding by assigning a continuous complexity score (0-1) to words. This finer-grained prediction is essential for generating more natural and effective text simplifications, as it allows for a more precise selection of replacement words and avoids oversimplification or misclassification of words. The authors highlight that previous methods often lacked a clear threshold for complexity, leading to errors in identifying or substituting complex words.

Methods

RS_GV employs a hybrid approach, combining traditional linguistic features with modern deep learning techniques. The core components include:

  • Hand-crafted Features: These are linguistically motivated features derived from various sources, including word properties (length, syllables), morphological information (prefixes, suffixes), psycholinguistic data (familiarity, age of acquisition), and WordNet relations (synonyms, hypernyms).
  • Sense-Relative Normalization: A key innovation is normalizing hand-crafted features based on the target word's senses in WordNet. This addresses the issue that a word's complexity can vary significantly depending on its meaning, allowing for a more contextually relevant feature representation.
  • Contextualized Embeddings: The system utilizes pre-trained word and character embeddings, specifically Flair (contextualized character embeddings), ELMO, and BERT (contextualized word embeddings), and GloVe (non-contextualized word embeddings). The authors found that stacked Flair embeddings, combining forward and backward versions with GloVe, performed best.
  • Neural Network Regressor: A simple feed-forward neural network with linear layers is used for regression, taking concatenated hand-crafted and embedding features as input.

Datasets

The primary dataset used is the CompLex corpus, which contains 9,476 annotated instances across three domains: bible, political, and biomedical texts. Each instance includes a target word, its containing sentence, and a crowd-sourced complexity value normalized to a 0-1 scale. The data was split into training, trial, and test sets for the SemEval-2021 Task 1.

Key Insights and Ideas

  • Sense Matters: Normalizing hand-crafted features based on WordNet senses significantly improves performance compared to simple min-max normalization, underscoring the importance of word meaning in complexity prediction.
  • Character Embeddings Shine: Contextualized character embeddings (Flair) outperformed other embedding types, including BERT and ELMO, suggesting their strong ability to capture nuanced word complexity.
  • Cross-Domain Advantage: Contrary to some previous findings, a cross-domain training approach yielded better results than within-domain training, possibly due to the limited size of the individual domain datasets for training a robust neural network.
  • Complexity is Subjective: The presence of instances with the same token in the same sentence but varying complexity values highlights the inherent subjectivity of LCP, making it a challenging task for machines.
  • Predicting Extremes is Hard: The system struggled with predicting the complexity of both very complex and very simple words, potentially due to insufficient training data for these extreme ranges.

Key Results

  • RS_GV achieved a Pearson correlation coefficient of r=0.7478 on the test set for the within-domain approach and r=0.7316 for the cross-domain approach.
  • The system ranked 34th out of 54 participants in the SemEval-2021 Task 1.
  • The best performing system achieved r=0.7886.
  • RS_GV performed best on biomedical terms and struggled most with europarl texts.
  • The system showed a tendency to predict complex words better than easy words, though it performed poorly on both extremes of the complexity spectrum.
  • The neural network approach significantly improved LCP compared to a linear regression baseline using only hand-crafted features.

Limitations and Future Work

  • Data Scarcity: The authors suggest that the dataset might be too small, especially for within-domain training and for accurately predicting very simple or very complex words.
  • Linearity Assumption: The lack of clear linearity between individual features and complexity might explain why the system couldn't match the top performers.
  • Subjectivity: The inherent subjectivity of LCP, as evidenced by varying annotations for similar contexts, poses a fundamental challenge.

Future work includes:

  • Fine-tuning embeddings on the specific dataset or using domain-specific pre-trained embeddings (e.g., BioFlair).
  • Exploring more hand-crafted features or refining existing ones, particularly sentence readability formulas.
  • Investigating more complex neural network architectures, such as convolutional layers, for improved feature selection.

Summary updated

BibTeX
@inproceedings{stodden2021rs,
  author = {Regina Stodden and Gayatri Venugopal},
  title = {{RS\_GV at SemEval-2021 Task 1: Sense Relative Lexical Complexity Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure