C3SL at SemEval-2021 Task 1: Predicting Lexical Complexity of Words in Specific Contexts with Sentence Embeddings

Raul Almeida, Hegler Tissot, Marcos Didonet Del FabroACL 2021

TL;DR

This paper explores whether pre-trained sentence embeddings (specifically Sent2Vec) can effectively predict the lexical complexity of words within their specific contexts. The authors found that while their approach was sound, pre-trained embeddings alone struggled to capture nuanced lexical complexity, especially across different domains, suggesting that context-specific fine-tuning or alternative embedding strategies might be necessary.

What Problem Does This Paper Address?

Understanding the lexical complexity of words in context is crucial for various Natural Language Processing (NLP) tasks, such as text simplification and improving reading comprehension for diverse audiences. The SemEval-2021 Lexical Complexity Prediction (LCP) shared task aimed to address this by providing a new English dataset annotated with a Likert scale for word complexity. The challenge lies in developing models that can accurately gauge how difficult a word is to understand given the surrounding text, moving beyond simple word frequency or general difficulty metrics.

Methods

The authors' approach involves a multi-step pipeline:

  1. Text Preprocessing: Sentences are cleaned by converting to lowercase, removing punctuation, and filtering out common "stop words" (unless they are the target word).
  2. Chunking and Embedding: The preprocessed sentence is split into four parts: the full sentence (S), the text before the target word (S₀), the target word(s) itself (T), and the text after the target word (S₁). Each of these chunks is then embedded using a pre-trained Sent2Vec model.
  3. Embedding Reduction: Instead of directly concatenating these embeddings, the authors compute the distance (using L2 norm) between the target word's embedding (T) and the embeddings of the other sentence parts (S, S₀, S₁). A boolean indicating if the target word embedding was null (due to being out-of-vocabulary for the pre-trained model) is also included. This creates a compact feature vector.
  4. Complexity Estimation: A Multi-Layer Perceptron (MLP) regressor from Scikit-Learn is trained on these reduced feature vectors to predict the lexical complexity score.

Datasets

The primary dataset used is the new annotated English dataset for the SemEval-2021 Lexical Complexity Prediction (LCP) shared task (Task 1). This dataset provides sentences with target words and their corresponding complexity scores on a Likert scale. The paper does not introduce a new dataset but evaluates on the one provided for the task.

Key Insights and Ideas

  • Pre-trained Embeddings Aren't Enough (Alone): The core finding is that general-purpose, pre-trained sentence embeddings like Sent2Vec, while capturing semantic relationships, do not inherently encode lexical complexity in a way that is directly transferable to predicting it across different domains.
  • Context Matters, But How Embeddings Capture It is Key: The approach of using sentence chunks and distances aims to leverage context. However, the results suggest that the way Sent2Vec represents semantic information might not align with how lexical complexity manifests, especially when applied to domains different from its training data.
  • The "i+1" Hypothesis Analogy: The authors draw a parallel to Stephen Krashen's Input Hypothesis in language acquisition, where learners understand content slightly above their current level. Their method of using distances from context chunks to the target word can be seen as an attempt to quantify how "obtainable" the target word is from its context, akin to the "+1" element.

Key Results

The paper reports results for two subtasks: single-word complexity prediction and multi-word expression complexity prediction. The authors' team, C3SL, achieved a Pearson correlation of 0.4598 and a Spearman correlation of 0.3983 for Subtask 1, placing them in the middle of the pack. For Subtask 2, C3SL achieved a Pearson correlation of 0.3941 and a Spearman correlation of 0.3675, also placing them in the middle. While their approach was functional, it did not outperform top-performing teams, which likely employed domain-specific fine-tuning or more sophisticated embedding strategies.

Limitations and Future Work

The authors acknowledge that their approach's main limitation is the inability of pre-trained Sent2Vec models to capture semantic complexity across domains. They suggest that future work could involve:

  • Domain-Specific Fine-Tuning: Training or fine-tuning Sent2Vec models on domain-specific corpora to better capture relevant semantic nuances.
  • Reinforcement Strategies: Exploring reinforcement learning techniques to enrich semantic information captured by sentence embeddings for classification tasks.
  • Alternative Embedding Reduction: Investigating different methods for reducing embedding representations beyond simple norm distances.

Summary updated

BibTeX
@inproceedings{almeida2021c,
  author = {Raul Almeida and Hegler Tissot and Marcos Didonet Del Fabro},
  title = {{C3SL at SemEval-2021 Task 1: Predicting Lexical Complexity of Words in Specific Contexts with Sentence Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure