cs60075_team2 at SemEval-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora

Abhilash Nandy, Sayantan Adak, Tanurima Halder, Sai Mahesh PokalaACL 2021

TL;DR

This paper tackles the challenge of predicting lexical complexity (how difficult a word or phrase is to understand) by fine-tuning various pre-trained Transformer language models on different text corpora. Their key insight is that aggregating predictions from multiple models, each trained on diverse data, significantly improves performance, mimicking the varied backgrounds of human annotators.

What Problem Does This Paper Address?

Readability is a crucial aspect of text comprehension. Complex words and phrases can significantly hinder understanding, making it difficult for readers to grasp the intended meaning. Lexical simplification aims to address this by replacing complex terms with simpler ones. A fundamental step in lexical simplification is accurately predicting the complexity of words and multi-word expressions (MWEs). Traditional methods often relied on hand-crafted features, but this paper explores how modern, powerful Transformer language models, pre-trained on vast amounts of text, can be leveraged for this task. The challenge lies in effectively adapting these general-purpose models to the specific nuances of lexical complexity prediction.

Methods

The core of this work involves fine-tuning pre-trained Transformer language models (specifically BERT and RoBERTa) for the Lexical Complexity Prediction (LCP) task. The authors experimented with several Transformer models, each pre-trained on different text corpora. These corpora ranged from general domains like Wikipedia and BooksCorpus to more specific ones such as the European Parliament Proceedings (Europarl), a bio-medical corpus (CRAFT), and religious texts (Bible).

The fine-tuning process involved feeding the model a sentence along with the target word or MWE, and the model was trained to output a complexity score between 0 and 1. The loss function used was Mean Squared Error (MSE), treating the problem as a regression task. A key aspect of their approach is aggregation: they combined the predictions from multiple fine-tuned models using two strategies: sample-wise average and sample-wise maximum. This aggregation was performed across various combinations of models, including those pre-trained on diverse domains, to find the optimal setup.

Datasets

The primary dataset used for fine-tuning and evaluation is the CompLex Dataset, which was specifically extracted from the corpora used for pre-training. The paper does not introduce a new dataset but rather leverages existing ones for both pre-training and fine-tuning.

Key Insights and Ideas

  • The Power of Aggregation: Combining predictions from multiple Transformer models, especially those trained on diverse corpora, significantly outperforms individual models. This suggests that different pre-training corpora equip models with distinct linguistic and contextual knowledge, and aggregating these diverse perspectives leads to a more robust complexity prediction.
  • Mimicking Human Annotators: The authors hypothesize that aggregating models trained on varied data effectively mimics the diverse backgrounds and prior knowledge of human annotators who created the CompLex Dataset. This is a crucial insight for understanding why aggregation works so well.
  • Domain Diversity Matters: Models pre-trained on domains related to, and even unrelated to, the CompLex Dataset (e.g., financial, legal data) contributed positively when aggregated. This indicates that general linguistic understanding gained from broad pre-training is highly transferable.
  • Average vs. Maximum Aggregation: The paper consistently found that sample-wise average aggregation performed better than sample-wise maximum aggregation across most combinations. This suggests that averaging out individual model predictions provides a more stable and accurate final score.

Key Results

The proposed method achieved strong results on the SemEval-2021 Task 1 for Lexical Complexity Prediction. The best performing model, achieved through aggregation, yielded a Pearson Correlation of 0.784 for single words and 0.836 for multi-word expressions. These results were superior to the baselines, which included XGBoost models utilizing hand-crafted features and GloVe embeddings. Notably, fine-tuning a single BERT-BASE-UNCASED model also showed competitive performance, especially for single words, highlighting the inherent power of large pre-trained models.

Limitations and Future Work

The paper acknowledges that while aggregation of Transformer models yields high performance, it also increases computational cost. The authors suggest that further exploration into optimal model selection and aggregation strategies could lead to more efficient yet equally effective solutions. Future work could also investigate other Transformer architectures and pre-training objectives, as well as explore different methods for combining model predictions. The paper also notes that the Pearson Correlation and Mean Squared Error are negatively correlated, as expected, but further analysis into the interplay of these metrics could be beneficial.

Summary updated

BibTeX
@inproceedings{nandy2021cs,
  author = {Abhilash Nandy and Sayantan Adak and Tanurima Halder and Sai Mahesh Pokala},
  title = {{cs60075\_team2 at SemEval-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure