archer at SemEval-2021 Task 1: Contextualising Lexical Complexity

Irene RussoACL 2021

TL;DR

This paper tackles the challenge of predicting how complex a word is within a specific sentence, moving beyond static word properties. By combining linguistic features with insights from pre-trained language models, the authors developed a system for the SemEval-2021 Lexical Complexity Prediction task, achieving a respectable rank. Their findings highlight the continued importance of word frequency while suggesting that in-context modeling, particularly with advanced language models, holds potential for future improvements.

What Problem Does This Paper Address?

Understanding and predicting lexical complexity is crucial for various Natural Language Processing (NLP) tasks, such as improving text readability and creating simplified versions of texts. Traditionally, lexical complexity has been assessed based on static properties of words, like their length, frequency, or age of acquisition. However, these "out-of-context" measures don't fully capture how a word's complexity can be influenced by the surrounding sentence. For instance, a common word might become more difficult to understand if used in a specialized or metaphorical sense within a particular context. This paper addresses the gap of effectively modeling this in-context lexical complexity, aiming to provide a more nuanced understanding of word difficulty.

Methods

The authors developed a system for the SemEval-2021 Lexical Complexity Prediction task, focusing on predicting a continuous complexity score for single words. Their approach combines two main types of features:

  • Out-of-Context Features: These are static properties of the target word's lemma, including:
  • Linguistic features: Length (characters), syllables.
  • Frequency-based features: Frequency in general corpora (Exquisite Corpus), children's movie subtitles, and the Visual Genome dataset.
  • Psycholinguistic features: Age of Acquisition (AoA) from the Kuperman et al. dataset.
  • Lexical features: Number of senses in WordNet, Scrabble score.
  • Domain indicators: Boolean features for the Bible, Europarl, and biomedical domains.
  • Other features: Presence in ImageNet, uppercase letter count (for acronym detection).
  • In-Context Features: These features aim to capture the influence of the surrounding text:
  • Language Model Position: The position of the target word among the top ten most probable words predicted by various pre-trained language models (BERT, XLNet, BART, ELECTRA, RoBERTa) in a masked word prediction task.
  • Previous Token Complexity: An average of the out-of-context complexity scores of the five preceding content words, calculated using a separate random forest regressor.

These features were fed into two regression models: a Random Forest (RF) regressor and an XGboost regressor, both implemented using sklearn. For the RF, the mean absolute error was used to measure split quality, and features were normalized.

Datasets

The primary dataset used is from the SemEval-2021 Lexical Complexity Prediction (LCP-2021) shared task. This dataset features English sentences annotated with a 5-point Likert scale for word complexity, encouraging annotators to consider the sentence context. The dataset comprises approximately 7,662 sentences and 3,298 unique tokens, extracted from three domains: the Bible, the European Parliament proceedings (Europarl), and a biomedical corpus. The paper notes that the biomedical domain contains the most complex target words and exhibits significant variability.

Additional datasets used for feature extraction include:

  • Exquisite Corpus: For general word frequency.
  • Children movies subtitles: For children's frequency.
  • Visual Genome descriptions corpus: For visual context frequency.
  • Kuperman et al. (2012) dataset: For Age of Acquisition (AoA).
  • WordNet: For the number of senses.

Key Insights and Ideas

  • Frequency Reigns Supreme (Still): Word frequencies, particularly those from specialized corpora like children's movies subtitles, remain highly predictive of lexical complexity. This reinforces the intuition that words encountered more often, especially in simpler contexts, are perceived as less complex.
  • In-Context Modeling's Limited Impact (So Far): Despite efforts to incorporate contextual information using pre-trained language models, these in-context features did not significantly improve performance on the trial sets. This suggests that while context is conceptually important, effectively translating it into predictive features for this specific task remains challenging.
  • The Nuance of Context: The paper highlights that even the same lemma can have different complexity scores in different sentences (e.g., "livers"). This variability underscores the need for context-aware models.
  • Acronyms are Difficult: The "Uppercase" feature, used to detect acronyms, was found to correlate with higher complexity ratings, aligning with intuition.
  • Domain Matters: Different domains (Bible, Europarl, biomedical) exhibit varying levels of lexical complexity, with the biomedical domain being the most challenging.

Key Results

The authors' system ranked 22nd out of 54 participants in the SemEval-2021 Lexical Complexity Prediction task for single words.

  • On the trial set, the Random Forest (RF) regressor generally performed slightly better than XGboost.
  • When using only out-of-context features, the RF achieved a Mean Absolute Error (MAE) of 0.063 and a Pearson correlation (R) of 0.817.
  • When including in-context features, the MAE increased to 0.095 with a lower R of 0.415, indicating a performance degradation on trial sets.
  • However, after testing with different trial sets, the authors inferred that in-context features could improve performance and included all features for the final test set.
  • The best result on the official test set was achieved using all features (both in- and out-of-context) with the Random Forest regressor, yielding a Pearson correlation of 0.7561 and an MAE of 0.0641.
  • Frequency features (Word freq + children freq, Word freq + VG freq) showed strong performance, with MAEs around 0.068-0.069 and R values around 0.74-0.752.

Limitations and Future Work

  • In-Context Feature Effectiveness: The authors acknowledge that the in-context features, particularly those derived from language model positions, did not yield the expected performance gains on trial sets. They suggest that these features might be beneficial with different trial set compositions or further tuning.
  • Future Analysis of Context: The paper proposes future work to explore how in-context target word complexity changes based on the frequencies of surrounding words. This suggests a deeper dive into the interaction between local context and word difficulty.
  • Sub-task 2 (Multi-word Expressions): The system was developed only for sub-task 1 (single words) and not for sub-task 2 (multi-word expressions).
  • Language Model Integration: While language models were used to generate features, their direct integration or fine-tuning for the complexity prediction task itself was not explored in depth. Future work could investigate more sophisticated ways of leveraging these models.

Summary updated

BibTeX
@inproceedings{russo2021archer,
  author = {Irene Russo},
  title = {{archer at SemEval-2021 Task 1: Contextualising Lexical Complexity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure