ANDI at SemEval-2021 Task 1: Predicting complexity in context using distributional models, behavioural norms, and lexical resources

Armand RotaruACL 2021

TL;DR

This paper tackles the challenge of automatically predicting how complex a word or phrase is within its surrounding text. The authors demonstrate that combining traditional psycholinguistic data with modern deep learning language models yields the best results, offering practical applications for improving text accessibility.

What Problem Does This Paper Address?

Understanding the complexity of language is crucial for making information accessible to a wider audience, including second language learners and individuals with low literacy. While previous research has focused on word complexity in isolation, real-world language use is always contextual. This paper addresses the gap by focusing on predicting contextualized lexical complexity, a more realistic and challenging task. The authors aim to develop models that can automatically assess how difficult a word or multi-word expression (MWE) is to understand when embedded within a sentence.

Methods

The researchers employed a multi-pronged approach, combining three main categories of predictors:

  • Behavioral Norms and Lexical Resources: This includes a rich set of psycholinguistic features like concreteness, imageability, age of acquisition, familiarity, emotional dimensions (valence, arousal, dominance), sensorimotor dimensions, lexical decision times, frequency counts, and CEFR word lists. These are established measures of word properties.
  • Context-Independent Distributional Models: These models, such as Skip-gram, GloVe, and ConceptNet NumberBatch, generate a fixed vector representation for each word, regardless of its context. They are trained on massive text corpora and have shown success in predicting various psycholinguistic variables.
  • Context-Dependent Distributional Models: This category includes powerful transformer-based models like BERT, RoBERTa, ELECTRA, ALBERT, and DeBERTa. These models generate word representations that are sensitive to the surrounding context, allowing for nuanced understanding.

The core innovation lies in how these predictors are combined. For each feature (e.g., familiarity), they create multiple predictors by considering the word itself, its context, and their interactions. For instance, they calculate the value of a feature for a word, for its context, and then combine these values using element-wise multiplication and absolute difference. This allows the model to capture not just the inherent properties of a word but also how its meaning and complexity are modulated by its surroundings. Finally, these diverse features are fed into regression models (ridge regression and gradient-boosted decision trees) to predict complexity ratings.

Datasets

The primary dataset used is CompLex 2.0, a newly compiled corpus for lexical complexity prediction. It comprises passages from the Bible, European Parliament proceedings, and biomedical journal articles. The dataset includes:

  • Training data: 7,662 single words and 1,517 multi-word expressions (MWEs).
  • Test data: 917 single words and 184 MWEs.

This dataset is notable for providing contextualized complexity ratings, differentiating it from previous work that often relied on isolated word ratings.

Key Insights and Ideas

  • Context is King (but not exclusively): While context-dependent models (like BERT) are powerful, context-independent models (like Skip-gram) surprisingly outperformed them in isolation. The authors attribute this to the sheer scale of data used to train the context-independent models.
  • Synergy of Predictors: Combining all three predictor types (behavioral norms, context-independent, and context-dependent models) yielded the best performance, though the improvement over simpler combinations was marginal. This suggests a high degree of redundancy in the information captured by these different approaches.
  • MWEs are Easier to Predict: Interestingly, the models performed better on multi-word expressions than on single words, even though the single-word dataset was significantly larger. The authors hypothesize that MWEs might have more constrained meanings, making their complexity easier to pin down.
  • Regularization is Less Critical Than Expected: For the combined predictor set, the degree of regularization (a technique to prevent overfitting) didn't significantly impact performance, suggesting the models were robust.

Key Results

The paper reports strong performance on the SemEval-2021 Task 1. Their best models achieved:

  • 4th rank in the single word sub-task with a Pearson correlation of r = .78 and MAE of .064.
  • 6th rank in the multi-word expression sub-task with a Pearson correlation of r = .85 and MAE of .067.

The figures in the paper visually demonstrate that combining predictors generally leads to higher Pearson correlations with actual complexity ratings across various regularization strengths.

Limitations and Future Work

The authors acknowledge that the improvement gained by combining all predictor classes was relatively small. They also note that context-independent models, despite their strong performance, might be compensated by their impressive vocabulary size and ease of use in practical applications.

Future work could explore more sophisticated ways to combine the different predictor types, potentially leading to further performance gains. Investigating the specific linguistic factors that contribute most to contextual complexity would also be a valuable direction.

Summary updated

BibTeX
@inproceedings{rotaru2021andi,
  author = {Armand Rotaru},
  title = {{ANDI at SemEval-2021 Task 1: Predicting complexity in context using distributional models, behavioural norms, and lexical resources}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure