LCP-RIT at SemEval-2021 Task 1: Exploring Linguistic Features for Lexical Complexity Prediction

Abhinandan Tejalkumar Desai, Kai North, Marcos Zampieri, Christopher HomanACL 2021

TL;DR

This paper presents LCP-RIT's approach to predicting lexical complexity, a task focused on identifying words that might be difficult for readers to understand. They explored various linguistic features, finding that psycholinguistic features like age of acquisition and prevalence significantly improve prediction accuracy over simpler statistical features.

What Problem Does This Paper Address?

Lexical complexity prediction (LCP) aims to quantify how difficult a word is to understand within a given context. This is crucial for applications like creating easier-to-read texts for language learners, children, or individuals with reading disabilities. While previous work often treated complexity as a binary (complex/not complex) classification, this paper tackles LCP as a regression problem, predicting a continuous complexity score from 0 to 1. This allows for a more nuanced understanding of word difficulty. The authors specifically address Sub-task 1 of SemEval-2021, focusing on predicting the complexity of single words.

Methods

The LCP-RIT team employed a feature engineering approach using a Random Forest Regressor (RFR) from scikit-learn. They experimented with a range of linguistic features to see which ones best predict word complexity.

Their approach involved:

  • Baseline Features: These included fundamental statistical features like word length, word frequency (based on Zipf's Law, where less frequent words are often longer and more complex), and syllable count (more syllables can indicate higher complexity). Character trigrams were also included as a baseline.
  • Psycholinguistic Features: These are core to their contribution and include:
  • Age of Acquisition (AoA): The age at which a word's meaning is typically learned.
  • Concreteness: How directly a word refers to a perceptible entity.
  • Familiarity: How well-known a word is to an individual.
  • Prevalence: The percentage of people who know a word.
  • Arousal: The degree to which a word's meaning evokes an active or passive emotional/physical response.
  • Additional Features: Part-of-speech (POS) tags and prior complexity labels from previous CWI tasks were also investigated.

They systematically added features to a baseline model to evaluate their individual and combined impact on performance.

Datasets

The primary dataset used is CompLex, an augmented version of an English multi-domain dataset. It contains sentences from various sources (Bible, European Parliament proceedings, biomedical texts) where words and multi-word expressions are annotated with a five-point Likert scale for complexity. For SemEval-2021 Task 1, the dataset was split into:

  • Training Data: 7,662 instances.
  • Test Data: 917 instances. 20% of the training data was also used for system development and testing.

Key Insights and Ideas

  • Psycholinguistic Features Shine: The study strongly suggests that psycholinguistic features are highly effective for LCP. Specifically, AoA, prevalence, and concreteness provided the most significant improvements.
  • Prevalence is Key: Prevalence, in particular, emerged as a very strong predictor, outperforming other features in reducing error and increasing correlation scores. This might be due to how it's calculated (percentage of people knowing a word) compared to familiarity (self-reported).
  • Redundancy of Prior Labels: Prior binary complexity labels from CWI tasks proved to be less effective in this regression setting. This highlights the limitations of binary classification for nuanced complexity prediction and suggests that the regression approach is more suitable.
  • Arousal's Limited Impact: While novel for LCP, arousal didn't significantly improve performance, suggesting its direct impact on perceived word complexity might be less pronounced than other psycholinguistic factors.
  • POS Tags are Weak Predictors: Part-of-speech tags showed minimal improvement and even a slight decrease in performance for one metric, indicating that grammatical category alone isn't a strong indicator of lexical complexity.
  • Larger Datasets Matter: The authors note that newer, larger psycholinguistic datasets (like those from Brysbaert et al.) are more beneficial than older ones (like MRC) due to their superior coverage.

Key Results

The LCP-RIT system, using a combination of psycholinguistic and baseline features, achieved strong performance on the training dataset:

  • Mean Absolute Error (MAE): 0.067
  • Mean Squared Error (MSE): 0.007
  • Pearson Correlation (R): 0.779
  • Spearman Correlation (ρ): 0.724

This performance surpassed their baseline model (statistical features + trigrams) by a notable margin. For instance, their MAE was 0.008 lower than the baseline.

On the test dataset, their performance was slightly lower but still competitive:

  • MAE: 0.072
  • MSE: 0.009
  • R: 0.709
  • ρ: 0.653

Limitations and Future Work

  • Overfitting Concern: The authors acknowledge that the reduced performance on the test set compared to the training set may indicate overfitting.
  • Transformer Models: They suggest that state-of-the-art transformer models could potentially achieve even better performance than their regression-based approach.
  • Cross-Lingual and Multilingual LCP: They express interest in exploring the performance of these features in languages other than English and on multilingual datasets.
  • Exploring Other Features: The authors plan to investigate other linguistic features described in related work (Shardlow et al., 2021).

Summary updated

BibTeX
@inproceedings{desai2021lcp,
  author = {Abhinandan Tejalkumar Desai and Kai North and Marcos Zampieri and Christopher Homan},
  title = {{LCP-RIT at SemEval-2021 Task 1: Exploring Linguistic Features for Lexical Complexity Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure