TUDA-CCL at SemEval-2021 Task 1: Using Gradient-boosted Regression Tree Ensembles Trained on a Heterogeneous Feature Set for Predicting Lexical Complexity

Sebastian Gombert, Sabine BartschACL 2021

TL;DR

This paper presents a system for predicting lexical complexity of words and multiword expressions using gradient-boosted regression trees trained on a diverse set of features. The key finding is that contextualized string embeddings, particularly those from the flair-mix-backward model, were the most influential features for predicting lexical complexity.

What Problem Does This Paper Address?

The paper tackles the challenge of "lexical complexity prediction," which aims to assign a score to words or phrases indicating how likely they are to be understood by a non-native speaker. This has practical applications in areas like creating tailored reading materials for language learners, assisting in vocabulary acquisition, and helping authors gauge the difficulty of their text. While previous work has explored this problem, often framing it as a classification task, this paper focuses on a regression approach to predict a continuous complexity score.

Methods

The authors employed gradient-boosted regression tree ensembles, specifically using the CatBoost implementation, to build their predictive models. They trained two separate models: one for single words and another for bigram multiword expressions. The core of their approach lies in the "heterogeneous feature set" they engineered. This set is a rich combination of:

  • Linguistic Features: Including syntactic tags (XPOS, UPOS, dependency, named entity) and universal features.
  • Word Embeddings: A mix of static (GloVe, fastText) and contextualized (BiomedNLP-PubMedBERT, distilgpt2, distilbert-base-uncased, flair-mix-forward, flair-mix-backward) embeddings.
  • Psycholinguistic Norm Lexica: Data reflecting human ratings of words on dimensions like pleasantness, arousal, and dominance.
  • Corpus-based Word Frequencies: Including frequencies from movie subtitles (SUBTLEXus), Google Books (character bigrams), and CEFR-level literature (EFLLex).
  • Word Lists: Binary features indicating inclusion in lists of simple words (Ogden's Basic English) or academic words (Academic Word List).

For multiword expressions, they concatenated feature vectors of individual words.

Datasets

The primary dataset used for training and evaluation was the CompLex corpus. This corpus consists of sentences from various sources (World English Bible, Europarl, CRAFT) and is divided into two sub-corpora: one for single words and one for bigram multiword expressions. For each word/bigram, a continuous complexity score (derived from annotator judgments on a Likert scale) is provided. The CompLex corpus was partitioned into training, test, and trial sets.

Key Insights and Ideas

  • Ensemble Power: Gradient boosting, an ensemble method, proved effective for combining a wide array of features, outperforming purely neural models that require dense, homogeneous input.
  • Contextual Embeddings Reign Supreme: The most significant finding is the overwhelming importance of contextualized string embeddings (specifically flair-mix-backward and flair-mix-forward) for predicting lexical complexity. This suggests that capturing character-level nuances and distributional knowledge across words and sentences is crucial.
  • Feature Diversity Matters: While contextual embeddings were dominant, the inclusion of diverse features (linguistic, psycholinguistic, frequency) contributed to the overall performance, indicating that lexical complexity is a multifaceted phenomenon.
  • Character-Level Embeddings' Advantage: The authors hypothesize that the success of character-level embeddings might stem from their ability to encode more nuanced information and capture regularities that fixed-word embeddings might miss.

Key Results

The systems achieved respectable rankings in the SemEval-2021 Task 1:

  • Single Word Complexity: Ranked 15th out of 54 systems, with a Pearson's correlation coefficient of 0.7618.
  • Multiword Expression Complexity: Ranked 19th out of 37 systems, with a Pearson's correlation coefficient of 0.8190.

While not achieving top ranks, their performance was close to the best systems, demonstrating the viability of their approach. Notably, the systems rarely assigned complexity scores below 0.2, suggesting a potential limitation in identifying very simple words.

Limitations and Future Work

  • Identifying Very Simple Words: The systems tend to assign non-zero scores even for simple words, indicating a need for features that can better distinguish extremely low complexity.
  • Interpretability of Embeddings: The strong influence of contextual string embeddings is acknowledged as hard to interpret directly, suggesting a need for further research into why they are so effective.
  • Feature Engineering: The authors suggest that further hyperparameter tuning and the addition of more features could potentially close the gap with the top-performing systems.
  • Future Research: Investigating the relationship between lexical complexity and character distribution, and exploring features that can improve the identification of very simple words are highlighted as future directions.

Summary updated

BibTeX
@inproceedings{gombert2021tuda,
  author = {Sebastian Gombert and Sabine Bartsch},
  title = {{TUDA-CCL at SemEval-2021 Task 1: Using Gradient-boosted Regression Tree Ensembles Trained on a Heterogeneous Feature Set for Predicting Lexical Complexity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure