TL;DR
This paper presents DeepBlueAI's winning approach to SemEval-2021 Task 1: Lexical Complexity Prediction (LCP). They achieved top rankings by fine-tuning multiple pre-trained language models (PLMs) with various strategies and then ensembling their predictions using a linear regression model. This ensemble approach effectively leverages the strengths of diverse PLMs to accurately predict word complexity.
What Problem Does This Paper Address?
Reading comprehension is significantly impacted by lexical complexity. Predicting how complex a word is within its context (Lexical Complexity Prediction or LCP) is crucial for applications like automatic text simplification and intelligent tutoring systems. Unlike simpler tasks like Complex Word Identification (CWI), which only flags words as complex or not, LCP aims to assign a continuous score reflecting the degree of complexity. This is challenging because a word's complexity can vary greatly depending on its context, as illustrated by the example of "refuge" in different sentences. The SemEval-2021 LCP task specifically aimed to address this by predicting complexity scores for both single words and multi-word expressions.
Methods
The DeepBlueAI system employs a two-stage approach:
- Fine-tuning Diverse PLMs: The core of their method involves fine-tuning several state-of-the-art pre-trained language models (PLMs), including BERT, ALBERT, RoBERTa, and ERNIE. To maximize diversity among these models, they experimented with various hyperparameters (e.g., learning rate, dropout, number of hidden layers) and training strategies.
- Pseudo-Labelling: They used a model trained on labeled data to predict labels for unlabeled data, then combined these pseudo-labels with the original training set to retrain the model. This helps to expand the training data.
- Data Augmentation: They augmented the training data by incorporating data from the CWI 2018 dataset and, for subtask 2, by using the subtask 1 dataset. Additionally, they applied techniques like synonym replacement, random insertion, swap, and deletion to create more varied training examples.
- Stacking Ensemble: After obtaining multiple fine-tuned PLMs, they used a simple linear regression model to combine their predictions. This "stacking" approach learns weights for each PLM's output, allowing the ensemble to leverage the strengths of individual models and produce a more robust final prediction. The architecture involves prepending a "query sentence" (e.g., "The word 'river' from the corpus 'Bible'") to the context to guide the PLM's focus.
Datasets
The primary dataset used for this task was Complex, introduced by Shardlow et al. (2020) for SemEval-2021 Task 1. This dataset contains 8979 annotated contexts with 5166 unique words, rated on a 5-point Likert scale (transformed to a 0-1 range). It draws from three corpora: Bible, Europarl, and Biomedical, and includes annotations for both single words and multi-word expressions.
They also utilized data from the CWI 2018 shared task for data augmentation.
Key Insights and Ideas
- Ensembling is Key: The paper strongly suggests that combining predictions from multiple diverse PLMs, even those that might not perform optimally individually, leads to superior performance. The linear regression stacking mechanism effectively captures these diverse signals.
- Training Strategies Enhance Diversity: While individual PLMs with added training strategies (pseudo-labeling, data augmentation) sometimes showed a slight decrease in performance compared to their base versions (as seen in Table 2), their inclusion in the final ensemble was crucial. This highlights that diversity in the base models is more important for the ensemble than peak individual performance.
- Context is Paramount: The LCP task's difficulty lies in the context-dependent nature of word complexity. The system's architecture, which explicitly feeds context to PLMs and uses special tokens to delineate query and context, is designed to capture these nuances.
Key Results
The DeepBlueAI system achieved significant success in SemEval-2021 Task 1:
- Ranked first for Subtask 2 (multi-word expressions).
- Ranked second for Subtask 1 (single words).
The paper shows that their ensemble approach, particularly the linear regression stacking, outperformed simple averaging of predictions, demonstrating the effectiveness of learning optimal weights for combining diverse models. For instance, Figure 3 visually confirms the improvement gained by using the LR stacking mechanism over simple averaging for subtask 1.
Limitations and Future Work
- Training Strategy Impact: The authors note that for some individual models, adding training strategies like pseudo-labeling and data augmentation slightly decreased performance (Table 2). While this was beneficial for the ensemble, it suggests a need for further research into optimizing these strategies for individual model performance.
- Subtask 2 Data Size: The paper mentions that the training dataset for subtask 2 was relatively small (1000 samples), leading them to augment it with subtask 1 data. Future work could explore more advanced techniques for handling limited data in subtask 2.
- Model Diversity: While they used heterogeneous PLMs, future work could explore even greater diversity by incorporating different model architectures or even non-PLM-based approaches into the ensemble.
Summary updated