CSECU-DSG at SemEval-2021 Task 1: Fusion of Transformer Models for Lexical Complexity Prediction

Abdul Aziz, Md. Akram Hossain, Abu Nowshed ChyACL 2021

TL;DR

This paper tackles the challenging task of predicting lexical complexity in text, which is crucial for applications like language learning and text simplification. The authors propose a novel approach by fusing the predictions of two powerful transformer models, BERT and RoBERTa, using a sentence-pair regression framework to achieve competitive results on the SemEval-2021 Task 1.

What Problem Does This Paper Address?

Lexical complexity prediction (LCP) aims to determine how difficult a word or phrase is to understand within a given sentence. This is a complex problem due to words having multiple meanings, intricate sentence structures, and the way words depend on each other. Accurately predicting LCP is vital for various Natural Language Processing (NLP) tasks, including making texts easier to understand for second language learners, individuals with dyslexia, or those with lower literacy levels. Existing methods struggle with the nuances of word usage and sentence context, making it difficult to assign a reliable complexity score.

Methods

The core of the proposed method lies in a sentence-pair regression framework built upon state-of-the-art transformer models. The authors fine-tune two pre-trained transformer models, BERT and RoBERTa, for this task.

Here's a breakdown of their approach:

  • Input Representation: For each sentence, they create pairs of the target word(s) and the sentence itself. These pairs are formatted as input sequences for the transformer models.
  • Pairwise Learning: The models are trained to predict a regression score for each sentence-word pair, essentially learning the contextual complexity.
  • Model Fusion: To leverage the strengths of both models, the individual regression scores predicted by BERT and RoBERTa are fused. This fusion is achieved by simply taking the arithmetic mean of their scores. This combined score represents the final predicted lexical complexity.
  • Transformer Models:
    • BERT (Bidirectional Encoder Representations from Transformers): A well-established model known for its strong performance on various NLP tasks, including sentence-pair regression.
  • RoBERTa (A Robustly Optimized BERT Pretraining Approach): An optimized version of BERT that removes the Next Sentence Prediction (NSP) objective and is trained with larger datasets and batch sizes, often leading to improved performance.

Datasets

The research utilizes the SemEval-2021 Task 1 dataset for lexical complexity prediction. This dataset is described as multi-domain and comprises texts from three different corpora:

  • Bible: Religious texts.
  • Europarl: European Parliament proceedings.
  • Biomed: Biomedical texts.

The task is divided into two subtasks:

  • Sub-task 1: Focuses on predicting the complexity of single words.
  • Sub-task 2: Focuses on predicting the complexity of multi-word expressions.

The paper provides detailed statistics on the training, validation, and test sets for both subtasks, including the distribution of instances across the three domains.

Key Insights and Ideas

  • Fusion is Key: Combining the predictions of BERT and RoBERTa significantly improved performance over using either model individually. This highlights the benefit of ensemble methods, even with closely related models.
  • Multi-word Complexity is Easier: The models consistently performed better on sub-task 2 (multi-word expressions) than on sub-task 1 (single words). This suggests that the richer contextual information available in multi-word phrases makes complexity estimation more straightforward for the models.
  • Transformer Power: The use of pre-trained transformer models like BERT and RoBERTa, fine-tuned for sentence-pair regression, proved effective for capturing the subtle nuances of lexical complexity.

Key Results

The CSECU-DSG system achieved competitive performance in the SemEval-2021 Task 1.

  • Sub-task 1 (Single Word Complexity): The proposed model achieved a Pearson correlation of 0.7716, ranking 9th overall.
  • Sub-task 2 (Multi-word Complexity): The model achieved a Pearson correlation of 0.8311, ranking 12th overall.

The paper also shows that the fused model's performance (0.7716 for sub-task 1 and 0.8311 for sub-task 2) was notably better than the individual BERT (0.7514 and 0.8077) and RoBERTa (0.7634 and 0.8211) models, demonstrating the effectiveness of their fusion strategy.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of LCP. However, the performance gap between single and multi-word complexity suggests that single-word complexity remains a more significant challenge.
  • Future Work: The authors plan to incorporate handcrafted features alongside state-of-the-art neural methods. This suggests a hybrid approach, aiming to further distill the complex relationships between sentences and words for improved complexity prediction.

Summary updated

BibTeX
@inproceedings{aziz2021csecu,
  author = {Abdul Aziz and MD. Akram Hossain and Abu Nowshed Chy},
  title = {{CSECU-DSG at SemEval-2021 Task 1: Fusion of Transformer Models for Lexical Complexity Prediction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure