Stanford MLab at SemEval-2021 Task 1: Tree-Based Modelling of Lexical Complexity using Word Embeddings

Erik Rozi, Niveditha Iyer, Gordon Chi, Enok Choe, Kathy J. Lee, Kevin Liu, Patrick Liu, Zander Lack, Jillian Tang, Ethan A. ChiACL 2021

TL;DR

This paper tackles the challenge of automatically predicting lexical complexity, a crucial step for making text accessible to a wider audience. The researchers developed a system using word embeddings and handcrafted features, achieving respectable rankings in a SemEval competition for both single-word and multi-word complexity prediction.

What Problem Does This Paper Address?

In today's information-saturated world, ensuring that content is accessible to everyone, regardless of their reading proficiency, is a significant challenge. Lexical complexity, or how difficult words are to understand, is a major barrier. This paper addresses the need for automated systems that can accurately assess this complexity. Such systems are vital for applications like text simplification, where complex words are replaced with simpler alternatives, or for tailoring content to specific reading levels. While there's a lot of text data available, very little is labeled for lexical complexity, making automated labeling a key bottleneck for developing these accessibility tools.

Methods

The Stanford MLab team employed a two-pronged approach for predicting lexical complexity:

  • Feature Engineering: They combined two types of features:
  • Embedding-based Features: They utilized pre-trained GloVe word embeddings to capture the semantic meaning of words. For single words, they applied PCA to reduce dimensionality. For context sentences, they computed the mean of word vectors after PCA.
  • Handcrafted Features: A set of manually designed features were included, such as the number of letters, capital letters, syllables, and digits in a token, as well as whether the token was capitalized (indicating a proper noun or technical term), the number of words in the context sentence, and the type of corpus the sentence originated from. For multi-word expressions, they concatenated the GloVe embeddings of individual words and their difference, followed by PCA.
  • Learning Models: They experimented with various regression models, including:
  • Linear Regression (as a baseline)
    • Ridge Regression
    • Support Vector Regression (SVR)
    • Decision Tree Regression
    • AdaBoost Regressor
    • XGBoost Regressor
    • LightGBM Regressor
    • Ensemble Stacking
    • Bagging Regressor
    • Fine-tuned BERT model

For the final submission, they selected the LightGBM Regressor for single-word complexity and also used it for multi-word complexity due to its strong performance.

Datasets

The research utilized data from the SemEval Task 1: Lexical Complexity Prediction competition. This dataset was an augmented version of the CompLex Corpus, which comprises English sentences from three genres: the Bible, Europarl, and biomedical writing. Tokens (both single-word and multi-word) were annotated for complexity on a scale of 1-5, which was then normalized to a continuous scale between 0 and 1.

Key Insights and Ideas

  • Contextual Variation vs. Intrinsic Properties: The study found that despite contextual variations, the inherent properties of a word (captured by GloVe embeddings) are largely sufficient to explain its lexical complexity. This suggests that a word's meaning and usage patterns are strong indicators of its difficulty.
  • Traditional Metrics Fall Short: Surprisingly, traditional text complexity metrics like Flesch-Kincaid Grade Level and SMOG Index showed very low correlation with the dataset's complexity values. This indicates that simple syllable and word counts are not as effective as semantic embeddings for predicting individual word complexity.
  • Handcrafted Features Add Value: While embeddings were powerful, the inclusion of handcrafted features (like syllable count and capitalization) provided marginal improvements, suggesting a synergistic effect between semantic and surface-level linguistic information.
  • LightGBM's Efficiency: The LightGBM Regressor emerged as a top performer, likely due to its efficiency in handling gradient boosting and its ability to manage complex feature interactions.

Key Results

The Stanford MLab system achieved the following rankings in the SemEval-2021 Task 1:

  • Single-word prediction: 27th out of 54 teams.
  • Multi-word prediction: 14th out of 37 teams.

For single-word complexity, the LightGBM Regressor achieved a Pearson correlation of 0.8056. For multi-word complexity, the same model achieved a Pearson correlation of 0.8280. The multi-word task performance was notably better, which the authors attribute to averaging GloVe representations of the two words in a token, providing more data points for the model.

Limitations and Future Work

  • Limitations:
    • The authors noted that Ensemble Stacking overgeneralized and did not perform as effectively as expected.
  • The BERT model, despite its contextual capabilities, did not perform as well as anticipated, only approaching baseline performance.
  • The paper acknowledges that while their model performed well on the CompLex dataset, its generalization to future, potentially different datasets is a subject for further investigation.
  • Future Work:
    • The researchers suggest incorporating additional contextual features, such as the proximity of neighboring words.
  • Exploring newer transformer models for representing contextual embeddings is also proposed.
  • The ultimate goal is to leverage these advancements to build systems that can help readers find suitable texts and tailor content to varied reading levels, thereby increasing accessibility.

Summary updated

BibTeX
@inproceedings{rozi2021stanford,
  author = {Erik Rozi and Niveditha Iyer and Gordon Chi and Enok Choe and Kathy J. Lee and Kevin Liu and Patrick Liu and Zander Lack and Jillian Tang and Ethan A. Chi},
  title = {{Stanford MLab at SemEval-2021 Task 1: Tree-Based Modelling of Lexical Complexity using Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure