CS-UM6P at SemEval-2021 Task 1: A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity

Nabil El Mamoun, Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, Ismail BerradaACL 2021

TL;DR

This paper introduces a deep learning system for predicting the complexity of words and multi-word expressions (MWEs) in text, a crucial step for making content more accessible. Their approach leverages pre-trained transformer encoders with a novel attention mechanism and uncertainty-weighted loss, achieving competitive results in the SemEval-2021 Lexical Complexity Prediction task.

What Problem Does This Paper Address?

Text Simplification (TS) aims to make written content easier to understand, benefiting individuals with reading difficulties, low literacy, or those learning a new language. A key sub-task of TS is Lexical Simplification (LS), which involves identifying complex words and replacing them with simpler synonyms while preserving meaning. This paper focuses on the Lexical Complexity Prediction (LCP) task, which is the first step in LS: assigning a difficulty score to a word or MWE. Previous work often treated LCP as a binary classification problem (complex/simple), but a recent dataset introduced a more nuanced 5-point Likert scale for fine-grained complexity prediction. The challenge lies in effectively modeling the contextual nuances that contribute to a word's perceived difficulty.

Methods

The proposed system builds upon pre-trained transformer encoder networks, specifically BERT and RoBERTa. The core of their approach involves:

  1. Transformer Encoder: They use either BERT or RoBERTa to generate contextualized word embeddings for the input sentence and the target word/MWE. The input is formatted by concatenating the context with the target word/MWE, separated by the encoder's special tokens (e.g., [SEP] or </s>).
  2. Residual Attention Block: On top of the transformer's output, an attention layer is applied. This layer learns to weigh the contextualized word embeddings based on their relevance to the complexity prediction task, effectively extracting salient features.
  3. Regression Module: The output from the attention block is concatenated with the pooled output of the transformer encoder (typically the embedding of the [CLS] or <s token). This combined representation is then fed into a simple regression module (one hidden layer, one output layer) to predict the complexity score.
  4. Loss Function: The model is trained to minimize a combined loss function consisting of:
  5. Root Mean Square Error (RMSE): A standard loss for regression tasks.
  6. Negative Pearson Correlation: An auxiliary loss designed to encourage the model to capture the correlation between predicted and true complexity scores.
  7. Uncertainty Loss Weighting: This technique, proposed by Kendall et al. (2017), automatically learns the optimal weights for combining the RMSE and auxiliary losses, avoiding manual tuning.

The authors investigate both single-task training (one model for word complexity, another for MWE complexity) and joint training (a single model for both).

Datasets

The study uses the dataset for the SemEval-2021 Task 1 (Lexical Complexity Prediction). This dataset is an augmented version of the Complex dataset, featuring:

  • Annotations for both single words and Multi-Word Expressions (MWEs).
  • Context sentences for each annotated item.
  • Complexity scores on a 5-point Likert scale, normalized to the range [0, 1].
  • Sentences drawn from three domains: Bible, EuroParl, and Biomedical texts.
  • Annotations aggregated from multiple annotators, ensuring at least four valid annotations per item.

The dataset consists of 7,662 samples for Sub-Task 1 (single word complexity) and 1,517 samples for Sub-Task 2 (MWE complexity) in the training set.

Key Insights and Ideas

  • Transformer Power for Complexity: Pre-trained transformers like BERT and RoBERTa are highly effective at capturing the contextual nuances required for lexical complexity prediction.
  • Attention Enhances Feature Extraction: The addition of an attention mechanism on top of transformer embeddings significantly improves the model's ability to focus on relevant contextual features for complexity assessment.
  • Joint Training Benefits: Training a single model for both word and MWE complexity prediction yields better overall correlation performances, suggesting that the tasks share complementary information.
  • Uncertainty Weighting Simplifies Optimization: Automatically learning the weights for combining different loss functions (RMSE and correlation) simplifies the training process and can lead to improved performance.

Key Results

The paper reports strong performance, with their system ranking among the top 10 submissions for both SemEval-2021 LCP sub-tasks.

  • Best Performer: The RoBERTa-large encoder combined with joint training achieved the best correlation metrics (Pearson and Spearman) for Sub-Task 1.
  • Sub-Task 1: For single word complexity, joint training with RoBERTa-large was superior for correlation. However, single-task training with RoBERTa-large yielded the best MAE, MSE, and R2.
  • Sub-Task 2: For MWE complexity, joint training with RoBERTa-large consistently provided the best results across all evaluation metrics.
  • Ablation Study: The ablation study confirmed that all components of their proposed model (attention block, auxiliary loss, uncertainty weighting) contribute positively to performance, with the auxiliary loss and uncertainty weighting showing particular benefits for correlation measures.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the performance trade-offs observed in the ablation study (e.g., auxiliary loss degrading MAE/MSE/R2). The performance differences between single-task and joint training, and between different encoders, highlight that model choice and training strategy are critical.
  • Future Work: The authors suggest that deep encoders provide better input representations, implying that exploring even larger or more advanced transformer architectures could further improve performance. The success of joint training also points towards exploring more sophisticated multi-task learning strategies.

Summary updated

BibTeX
@inproceedings{mamoun2021cs,
  author = {Nabil El Mamoun and Abdelkader El Mahdaouy and Abdellah El Mekki and Kabil Essefar and Ismail Berrada},
  title = {{CS-UM6P at SemEval-2021 Task 1: A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure