An Evaluation of Binary Comparative Lexical Complexity Models

Kai North, Marcos Zampieri, Matthew ShardlowNAACL 2022

TL;DR

This paper introduces "binary comparative lexical complexity prediction" (BC-LCP), a new approach to identifying complex words in text by directly comparing pairs of words within the same sentence. They developed a new dataset, CompLex-BC, and found that a traditional Support Vector Machine (SVM) model, using carefully selected linguistic features, outperformed more complex transformer-based models like BERT for this specific task.

What Problem Does This Paper Address?

Text simplification (TS) systems aim to make written content more accessible to a wider audience, including language learners and individuals with reading disabilities. A crucial first step in TS is identifying "complex words" that might hinder comprehension. Traditionally, this has been framed as a binary classification (is this word complex or not?) or regression task (how complex is this word on a scale?). However, the authors argue that a more nuanced approach is needed: directly comparing the complexity of two words within the same sentence. This "binary comparative" approach could significantly improve how TS systems select and rank substitute words, as it allows for more efficient data generation and a finer-grained understanding of relative complexity.

Methods

The core of the paper is the exploration of binary comparative lexical complexity prediction (BC-LCP). The authors experimented with three main modeling approaches:

  • Support Vector Machine (SVM): This model utilized a set of established linguistic features known to be indicative of lexical complexity. These features include word length, word frequency (from the British National Corpus), syllable count, average age of acquisition, and prevalence (familiarity). The SVM was trained using a Radial Basis Function (RBF) kernel.
  • BERT: A pre-trained BERT model (bert-base-uncased) was fine-tuned for the BC-LCP task. Various input encoding strategies were explored, including using only the target word, the target word with a single context sentence, and two contexts.
  • BERT + MLP: This hybrid model combined the contextual understanding of BERT with the word-level features used by the SVM. It aimed to leverage both types of information for improved performance.

Additionally, two naive baseline models, a random classifier (RC) and a majority classifier (MC), were used for comparison.

Datasets

  • CompLex 2.0: This existing dataset, used in SemEval-2021 Task 1, contains 9,000 instances of individual words with continuous complexity values derived from a 5-point Likert-scale annotation.
  • CompLex-BC: The authors created this new dataset by extracting 1,940 sentences from CompLex 2.0 that contained at least two identified complex words. For each sentence, they created pairs of these complex words and assigned a binary label indicating which word was considered more complex by the original CompLex 2.0 annotators.

Key Insights and Ideas

  • Direct Comparison is Powerful: The paper demonstrates that directly comparing word complexities within a sentence (binary comparative LCP) is a viable and effective approach, offering advantages over independent classification or regression.
  • SVM's Enduring Strength: Surprisingly, a well-feature-engineered SVM model outperformed transformer-based models like BERT on this specific BC-LCP task. This suggests that for certain tasks, especially with limited data, traditional methods can still be highly competitive.
  • Contextual vs. Word-Level Features: The authors found that for BC-LCP, word-level features (length, frequency, etc.) were more discriminative than contextual features provided by BERT. This is attributed to the high similarity of contextual information for two words within the same sentence, which can confuse BERT.
  • Data Augmentation Potential: The binary comparative approach allows for significant data augmentation. A dataset with continuous complexity values can be transformed into a much larger set of comparative instances, which is crucial for training complex models.

Key Results

The paper reports the following key performance metrics (weighted average F1-score):

  • SVM: 0.86
  • BERT + MLP: 0.80
  • BERT: 0.44
  • MC: 0.39
  • RC: 0.50

The SVM model achieved the highest F1-score, indicating superior performance in identifying which of two words is more complex. The BERT model's low performance suggests it struggled to converge, likely due to insufficient data and the nature of the comparative task.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the CompLex-BC dataset, while novel, is relatively small (1,940 instances). They believe more data is needed for transformer-based models to reach their full potential.
  • Contextual Information for BERT: The high similarity of contextual information for word pairs within the same sentence posed a challenge for BERT. Future work could explore more sophisticated ways to disentangle or utilize this similar context.
  • "Neutral" Class: The current binary comparative approach doesn't handle cases where two words have very similar complexity. The authors suggest exploring a "neutral" class for such scenarios.
  • Multilingual BC-LCP: The current work is limited to English. Investigating the feasibility of BC-LCP for other languages is a promising future direction.
  • Model Evaluation: The authors plan to explore different variables impacting BC-LCP, including evaluating models on target words in diverse contexts and incorporating a "neutral" class.

Summary updated

BibTeX
@inproceedings{north2022evaluation,
  author = {Kai North and Marcos Zampieri and Matthew Shardlow},
  title = {{An Evaluation of Binary Comparative Lexical Complexity Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure