UTFPR at SemEval-2021 Task 1: Complexity Prediction by Combining BERT Vectors and Classic Features

Gustavo Henrique PaetzoldACL 2021

TL;DR

This paper presents UTFPR's approach to the SemEval-2021 Lexical Complexity Prediction task, combining traditional linguistic features with powerful BERT embeddings. They found that while BERT vectors significantly boost performance, they are best used as a complement to established features rather than a standalone solution.

What Problem Does This Paper Address?

Accurately identifying complex words and phrases in text is crucial for applications like text simplification, making literature accessible to children or second-language learners, and improving the readability of official documents. Previous work on Complex Word Identification (CWI) often relied on ensemble methods with handcrafted linguistic features. However, these approaches predated the widespread adoption of transformer-based language models like BERT, which have revolutionized natural language processing. This paper addresses the gap by exploring how to effectively integrate BERT's contextual representations with traditional features for lexical complexity prediction.

Methods

The UTFPR team employed a multi-faceted approach, combining classic linguistic features with BERT embeddings. Their feature set includes:

  • Word/Phrase Frequency: Utilizing both simple word frequencies and n-gram frequencies across various context windows.
  • Word/Phrase Length: Measuring the number of characters in words or the average number of characters in phrase words.
  • Number of Senses: Leveraging WordNet to count the senses of words or the average number of senses for phrase words.
  • BERT Vector: Extracting 768-dimensional embeddings from the bert-base-uncased model. For out-of-vocabulary words or fragmented tokens, they averaged the representations of the fragments.

They experimented with several machine learning models, including Ridge Regression, SVM, AdaBoost, Gradient Boosting, and Random Forests. For phrase complexity, they investigated the principle of compositionality, testing how combining word-level features (frequency, length, senses) using averaging, maximum, and minimum functions impacted phrase complexity prediction.

Datasets

The primary dataset used is the CompLex dataset, an extended version of which was provided for the SemEval-2021 Lexical Complexity Prediction (LCP 2021) shared task. This dataset includes sentences, target words/phrases, and complexity scores derived from judgments by English speakers. The task was split into two sub-tasks: single words and phrases.

For feature extraction, specifically n-gram frequencies, they utilized the SubIMDB corpus, a collection of subtitles from children, family, and comedy movies and series.

Key Insights and Ideas

  • BERT as a Complement, Not a Replacement: The most significant finding is that BERT vectors, even when not specifically fine-tuned for the task, are a powerful addition to traditional features. They significantly improve performance but don't entirely supersede the utility of classic features like word frequency.
  • Compositionality for Phrases: The principle of compositionality, where phrase complexity is modeled based on the complexity of its constituent words, generally improves performance for phrases. Averaging word-level features proved to be a beneficial strategy.
  • N-gram Frequencies Can Be Detrimental: Surprisingly, discarding n-gram frequencies often led to improved model performance, particularly for single words. This suggests that the contextual information captured by BERT might be sufficient, or that n-gram features might introduce noise or redundancy in this specific task.
  • Conservative Predictions: The authors note that their systems tended to underestimate complex words and overestimate simple ones, leading to a "conservative nature" in their predictions, which likely impacted their overall ranking.

Key Results

The UTFPR systems achieved respectable, though not top-tier, results in the SemEval-2021 LCP task.

  • Single Words: Placed 45th out of 55 systems.
  • Phrases: Placed 29th out of 38 systems.

Their best-performing configuration for both sub-tasks was an SVM trained on word/phrase frequencies, length, number of senses, and BERT vectors, excluding n-gram frequencies. The Pearson correlation for the single words sub-track was 0.6875, and for phrases, it was 0.7601.

Limitations and Future Work

The authors acknowledge several limitations:

  • Unoptimized BERT: The BERT model was used in its original pre-trained form without task-specific fine-tuning. This is a major area for future work.
  • Conservative Predictions: The tendency to make conservative predictions (underestimating high complexity, overestimating low complexity) limited their ranking.

Future work will focus on:

  • Optimizing BERT: Fine-tuning BERT vectors specifically for the lexical complexity prediction task.
  • Exploring other Transformer Models: Investigating the efficacy of other advanced language models.
  • Further Compositionality Exploration: Deeper analysis into compositionality for phrases.

Summary updated

BibTeX
@inproceedings{paetzold2021utfpr,
  author = {Gustavo Henrique Paetzold},
  title = {{UTFPR at SemEval-2021 Task 1: Complexity Prediction by Combining BERT Vectors and Classic Features}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure