RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction

Gang Rao, Maochang Li, Xiaolong Hou, Lianxin Jiang, Yang Mo, Jianping ShenACL 2021

TL;DR

This paper introduces a novel contextual attention-based model using RoBERTa for predicting lexical complexity. By employing a two-stage fine-tuning approach and an ensemble strategy, the model achieved second place in the SemEval-2021 Task 1, demonstrating the effectiveness of contextual embeddings and attention mechanisms for this task.

What Problem Does This Paper Address?

The paper tackles the challenge of Lexical Complexity Prediction (LCP), which aims to assign a complexity score to individual words within a sentence. Unlike previous work that often treated LCP as a binary classification problem (complex vs. non-complex), this task, as defined by SemEval-2021 Task 1, treats complexity as a continuum on a 5-point Likert scale. The dataset is multi-domain, meaning words from various sources (like Bible, Europarl, Biomedical) are included, posing a challenge due to potential domain-specific complexities. The authors highlight that the distribution of complexity scores differs significantly between single-word and multi-word complexity datasets, suggesting the need for tailored approaches.

Methods

The core of the proposed model is built upon RoBERTa, a powerful pre-trained language model. The authors employ a two-stage fine-tuning strategy:

  1. Corpus Classification Fine-tuning: RoBERTa is first fine-tuned on a corpus classification task. This step aims to imbue the model with domain-specific knowledge before it tackles the LCP task. This is achieved by training RoBERTa for just one epoch on all available data (train, trial, and test), allowing it to learn general domain understanding.
  2. Contextual Attention-based LCP Model: After the initial fine-tuning, the model leverages RoBERTa's contextual embeddings. For each target word, an attention mechanism is used to weigh the embeddings of surrounding context words. This weighted context embedding is then combined with the target word's embedding. The model then uses a multi-layer perceptron (MLP) to predict the final LCP score.

Key technical details include:

  • Token Pooling: Average pooling of target word embeddings to create a target vector.
  • Attention Mechanism: Soft alignment between context token embeddings and the target vector to compute attention weights.
  • Weighted Combination: The final context vector is a weighted sum of context token embeddings, and the final target vector is a weighted sum of the last 12 layers' context and target vectors.
  • Gradual Unfreezing: A technique where initially only the head layers of the model are trained, followed by unfreezing all parameters.
  • Multi-Sample Dropout: Applied at the last layer for regularization.
  • Ensemble Strategy: K-fold cross-validation is used to train K models, and their predictions are ensembled to produce the final result.

Datasets

The primary dataset used is from SemEval-2021 Task 1, which is a multi-domain English dataset annotated with a 5-point Likert scale for lexical complexity. The dataset is divided into single-word and multi-word complexity tasks. The authors also mention using a corpus classification dataset derived from Bible, Europarl, and Biomedical sources for their initial fine-tuning step.

Key Insights and Ideas

  • Domain-Specific Fine-tuning Matters: Pre-training RoBERTa on a corpus classification task significantly improved performance on the LCP task compared to using the original RoBERTa model. This highlights the importance of domain knowledge for accurate lexical complexity prediction.
  • Contextual Attention is Crucial: The attention mechanism effectively captures the contextual nuances of words, which is vital for understanding their complexity. Simply using raw RoBERTa embeddings was less effective than incorporating this attention-based contextual representation.
  • Single vs. Multi-Task Models: The authors found that the distribution of complexity scores differed significantly between single and multi-word datasets, leading them to build separate models for each, which ultimately contributed to better overall performance.
  • Ensembling Boosts Performance: Combining predictions from multiple models trained via K-fold cross-validation provided a substantial performance improvement, demonstrating the robustness of the ensemble approach.

Key Results

The proposed contextual attention-based model with RoBERTa achieved the 2nd place in the SemEval-2021 Task 1 (sub-task 2) with a Pearson correlation of 0.8575. The paper also reports that the fine-tuned RoBERTa model for corpus classification achieved an accuracy of 0.99. For the single LCP task, the best ensemble model achieved a Pearson correlation of 0.7629.

Limitations and Future Work

  • Data Augmentation for Multi-Task: The authors acknowledge that the multi-task dataset is relatively small (1616 examples). They employed a data augmentation strategy by recalling single-task examples based on cosine similarity to enrich the multi-task training data.
  • Future Work: The authors plan to explore better models for text representation and further investigate data augmentation techniques for all tasks. They also suggest that future work could focus on improving the model's ability to utilize domain information more effectively.

Summary updated

BibTeX
@inproceedings{rao2021rg,
  author = {Gang Rao and Maochang Li and Xiaolong Hou and Lianxin Jiang and Yang Mo and Jianping Shen},
  title = {{RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure