LangResearchLab NC at SemEval-2021 Task 1: Linguistic Feature Based Modelling for Lexical Complexity

Raksha Agarwal, Niladri ChatterjeeACL 2021

TL;DR

This paper tackles the challenge of automatically identifying and scoring the complexity of words and phrases within a sentence, a crucial step for improving text readability and comprehension. The researchers developed a system that leverages a rich set of linguistic features, including lexical, semantic, and syntactic information, to predict a continuous complexity score. Their approach achieved competitive results on the SemEval-2021 Task 1, demonstrating the effectiveness of feature-based regression for this task.

What Problem Does This Paper Address?

The ability to understand text is fundamental, and complex words can significantly hinder comprehension. While previous work has focused on binary classification (complex vs. non-complex), this paper addresses the more nuanced task of Lexical Complexity Prediction (LCP). LCP aims to assign a continuous score (between 0 and 1) to a target word or phrase, indicating its degree of complexity. This is vital for applications like automatic text simplification, where identifying how complex a word is, not just if it's complex, allows for more targeted and effective simplification strategies. The challenge lies in capturing the multifaceted nature of lexical complexity, which is influenced by factors like word rarity, semantic depth, and syntactic role.

Methods

The core of the LangResearchLab NC system is a Random Forest Regressor trained on a comprehensive set of linguistic features. For Single Word Targets, the system extracts features related to:

  • Corpus Features: Which of the three main corpora the sentence originates from.
  • Shallow Features: Basic word properties like length, syllable count, vowel count, and part-of-speech (POS) tags (both simple and detailed).
  • NLTK WordNet Features: Information about word relationships, such as the number of hypernyms and morphemes.
  • Exquisite Corpus (EC) Features: Word frequency and Zipf frequency from a large, diverse corpus.
  • SUBTLEX Features: Word frequency and contextual diversity from the SUBTLEXus database.
  • Language Model (LM) Features: Perplexity and phrase scores derived from a trigram language model.
  • Character Language Model (CharLM) Features: Probability of the target word based on character trigrams.
  • Psycholinguistic Features: Age of acquisition, concreteness, imageability, and meaningfulness ratings from the MRC database.
  • Kucera and Francis (KF) Features: Written frequency and number of text categories the word appears in.
  • Ogden Feature: A binary feature indicating presence in Ogden's Basic English list.
  • Inquirer Tag Features: Binary features for each of the 182 General Inquirer categories, filtered for compatibility with POS tags.

For Multiword Targets, two strategies were explored:

  1. Single Word Combination: Predicting complexity for each word individually and then combining these scores using various fuzzy set operations (average, maximum, minimum, algebraic sum, product).
  2. Feature Combination: Concatenating features of individual words and training a regression model on this expanded feature space. The complexity score from the Single Word Target model was also included as a feature.

The authors also experimented with BERT embeddings and Neural Networks, but found that Random Forest Regressors with their linguistic features performed best.

Datasets

The dataset used for this task is an augmented version of CompLex (Shardlow et al., 2020). This dataset comprises sentences from three corpora:

  • World English Bible Translation
  • English Portion of the European Parliament proceedings
  • Articles from the CRAFT corpus (biomedical domain)

The dataset is split into Train, Trial, and Test subsets.

Key Insights and Ideas

  • Linguistic Features are Powerful: The study strongly suggests that a rich set of linguistic features, encompassing lexical, semantic, and syntactic information, is highly effective for predicting lexical complexity. The performance of Random Forest Regressors underscores this.
  • Exquisite Corpus Features are Crucial: Removing features from the Exquisite Corpus led to the most significant performance drop, highlighting the importance of broad frequency and Zipfian distribution information.
  • Context Matters, But is Hard to Capture: While the model performs well, error analysis indicates that distinguishing the complexity of proper nouns in different contexts remains a challenge, suggesting that deeper contextual understanding is an area for future improvement.
  • Multiword Complexity is Best Modeled by Combining Individual Word Features: The feature combination approach for multiword targets, where individual word features and their predicted complexities are concatenated, yielded superior results compared to simply combining individual word scores.
  • BERT Doesn't Always Trump Feature Engineering: Interestingly, the paper found that BERT embeddings alone did not improve performance over linguistic features, and even when combined with linguistic features in a neural network, they did not outperform the Random Forest Regressor. This suggests that for this specific task, carefully engineered linguistic features can be highly competitive.

Key Results

  • For Single Word Targets, the system achieved a Pearson correlation of 0.7402 on the test set using a Random Forest Regressor.
  • For Multiword Targets, the feature combination approach using Bagging on a Random Forest Regressor achieved a Pearson correlation of 0.8244 on the test set.
  • The system achieved 36th and 17th rank on the two subtasks (Single Word Target and Multiword Target, respectively) of the SemEval-2021 Task 1.
  • Error analysis showed that 87% of test samples had an absolute error less than 0.10.
  • Samples from the Biomedical corpus tended to have the highest errors.

Limitations and Future Work

  • Contextual Nuances: The authors acknowledge that their model struggles to capture the contextual differences in complexity for proper nouns, indicating a need for more sophisticated contextual modeling.
  • BERT Performance: While BERT is a powerful tool, its direct application in this specific task did not yield the best results, suggesting further research into how to best integrate or adapt transformer-based models for lexical complexity prediction.
  • Further Improvement: The authors suggest that future work could focus on improving the model's ability to capture contextual nuances and potentially explore more advanced deep learning architectures tailored for this task.

Summary updated

BibTeX
@inproceedings{agarwal2021langresearchlab,
  author = {Raksha Agarwal and Niladri Chatterjee},
  title = {{LangResearchLab NC at SemEval-2021 Task 1: Linguistic Feature Based Modelling for Lexical Complexity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure