ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution

Xuanming Zhang, Zixun Chen, Yu ZhouarXiv 2024 · DOI · arXiv

Abstract

Lexical Substitution discovers appropriate substitutes for a given target word in a context sentence. However, the task fails to consider substitutes that are of equal or higher proficiency than the target, an aspect that could be beneficial for language learners looking to improve their writing. To bridge this gap, we propose a new task, language proficiency-oriented lexical substitution. We also introduce ProLex, a novel benchmark designed to assess systems' ability to generate not only appropriate substitutes but also substitutes that demonstrate better language proficiency. Besides the benchmark, we propose models that can automatically perform the new task. We show that our best model, a Llama2-13B model fine-tuned with task-specific synthetic data, outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex.

Introduction

Lexical Substitution (LS) is a fundamental task in Natural Language Processing: given a target word in a specific context, the goal is to find a replacement that preserves the original meaning. While traditional LS is useful for general paraphrasing, it often fails to meet the needs of English Second Language (L2) learners. These learners frequently encounter a "vocabulary plateau," where they rely on simple, low-proficiency words (e.g., using "good" instead of "beneficial" or "advantageous") because they lack the confidence or knowledge to use more advanced synonyms.

Existing benchmarks like LS07 or Swords prioritize contextual appropriateness above all else. However, for a learner, a substitute that is merely appropriate but equally simple does not facilitate linguistic growth. This paper introduces a new paradigm: language proficiency-oriented lexical substitution, requiring substitutes to be both contextually correct and demonstrably more advanced based on standardized frameworks like the Common European Framework of Reference for Languages (CEFR).

Methodology

To address this new task, the researchers developed a multi-stage pipeline to construct the ProLex benchmark and train specialized models.

The multi-stage pipeline for ProLex construction, from initial word-sentence pair selection to final proficiency-oriented filtering.

As illustrated in Figure 1, the construction process involves five key steps:

  1. Selection: Extracting word-sentence pairs from the TOEFL-11 corpus of non-native essays.
  2. Grammar Correction: Using a fine-tuned GEC model to ensure the source sentences are grammatically sound.
  3. Candidate Generation: Leveraging GPT-4 to generate a broad pool of potential substitutes.
  4. Human Annotation: Expert linguists evaluate candidates for semantic preservation, natural collocation, and grammatical correctness.
  5. CEFR Filtering: A CEFR checker filters the validated list to ensure substitutes are of equal or higher proficiency than the target word.

To build models capable of this task, the authors fine-tuned Llama-2 and Vicuna models using synthetic data derived from GPT-4 ($D_L$) and adapted from the Swords benchmark ($D_S$).

The ProLex Benchmark

The resulting ProLex benchmark consists of 680 unique word-sentence pairs, featuring 1,955 acceptable substitutes and 1,559 proficiency-oriented substitutes. The benchmark is designed to reflect the pedagogical "Zone of Proximal Development," challenging learners just slightly beyond their current level.

Distribution of acceptable and proficiency-oriented substitutes across CEFR levels for low and medium proficiency learners.

Analysis of the benchmark (Figure 3) reveals that proficiency-oriented substitutes ($w^a_p$) tend to shift toward higher CEFR levels (B2, C1, C2) compared to the general pool of acceptable substitutes ($w^a$). This confirms that the benchmark successfully identifies words that can help learners "level up" their prose.

Key Results

The researchers evaluated several models using F-score at $k=10$. The results demonstrate that standard state-of-the-art LS systems (like BERT-LS) are insufficient for this task, as they lack the proficiency awareness required for pedagogical applications.

  • Llama2-13B Performance: When fine-tuned on the aggregate synthetic dataset ($D_{LS}$), the Llama2-13B model achieved an F-score of 54.2.
  • Competitive Edge: This specialized model outperformed ChatGPT (51.5) and reached parity with GPT-4 (54.7), proving that task-specific instruction tuning can match the performance of much larger, general-purpose models.
  • Domain Sensitivity: Models trained on learner-centric data ($D_L$) performed significantly better than those trained on native-speaker contexts ($D_S$), highlighting the importance of understanding the specific linguistic distribution of L2 writing.

Limitations and Future Work

While ProLex represents a significant step forward, it currently focuses on single-word substitutions. Future work will aim to expand the benchmark to include multi-word expressions and idioms, which are notoriously difficult for CEFR checkers to evaluate. Additionally, the authors plan to incorporate broader discourse context to ensure that advanced vocabulary suggestions remain stylistically appropriate for the overall essay, avoiding the pitfall of "purple prose."

Summary updated

Abstract

Lexical Substitution discovers appropriate substitutes for a given target word in a context sentence. However, the task fails to consider substitutes that are of equal or higher proficiency than the target, an aspect that could be beneficial for language learners looking to improve their writing. To bridge this gap, we propose a new task, language proficiency-oriented lexical substitution. We also introduce ProLex, a novel benchmark designed to assess systems' ability to generate not only appropriate substitutes but also substitutes that demonstrate better language proficiency. Besides the benchmark, we propose models that can automatically perform the new task. We show that our best model, a Llama2-13B model fine-tuned with task-specific synthetic data, outperforms ChatGPT by an average of 3.2% in F-score and achieves comparable results with GPT-4 on ProLex.
BibTeX
@misc{zhang2024prolex,
  author = {Xuanming Zhang and Zixun Chen and Yu Zhou},
  title = {{ProLex: A Benchmark for Language Proficiency-oriented Lexical Substitution}},
  year = {2024},
  eprint = {2401.11356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.11356},
}
Figure