New Evaluation Paradigm for Lexical Simplification

Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, Xiaoye OuyangarXiv 2025 · arXiv · OpenReview

TL;DR

Large Language Models (LLMs) can simplify sentences in a single step, bypassing traditional multi-stage lexical simplification (LS) pipelines. However, existing LS datasets are ill-suited for evaluating these LLM outputs. This paper introduces a novel human-AI collaborative method to create an "all-in-one" LS dataset and demonstrates that a multi-LLM approach significantly outperforms single-prompt LLMs and traditional methods.

What Problem Does This Paper Address?

Lexical Simplification (LS) aims to make text more accessible by replacing complex words with simpler alternatives. Traditionally, LS is a three-step process: identifying complex words (CWI), generating substitutes (SG), and ranking those substitutes (SR). Each step has its own datasets and evaluation metrics. However, the advent of powerful LLMs has changed the game. LLMs can now simplify entire sentences with a single prompt, rendering the traditional pipeline and its associated datasets insufficient for evaluation. Existing datasets often focus on providing substitutes for one complex word at a time, failing to capture the LLM's ability to simplify all complex words within a sentence and assess the overall sentence simplification. This gap hinders the development and evaluation of next-generation LS systems.

Methods

The authors propose a two-pronged approach:

  1. A Novel "All-in-One" LS Dataset Construction: They introduce a human-AI collaborative method to build a dataset that annotates all simplifiable complex words in a sentence and provides suitable substitutes for each. This process leverages existing CWI datasets to avoid redundant complex word annotation. Automated methods (including LLMs) generate initial substitute candidates, which human annotators then refine and validate. This approach aims to create a dataset suitable for evaluating end-to-end LS.
  2. Multi-LLM Collaboration for LS: They explore various LLM-based LS strategies:
  3. Single-Prompt LS: Using LLMs directly with carefully crafted prompts to simplify sentences.
  4. In-Context Learning & Chain-of-Thought (COT): Enhancing single-prompt performance by providing examples and guiding the LLM through step-by-step reasoning.
  5. Multi-LLM Collaboration (CoLLS): A novel framework where multiple LLMs emulate human expert teams to perform CWI, SG, and a new "Validation" step (which assesses simplicity and faithfulness at the sentence level). This approach uses majority voting among LLMs to improve robustness and accuracy.

Datasets

  • Existing CWI Datasets: Used as a foundation for complex word identification.
  • Newly Introduced "All-in-One" LS Dataset: Constructed through human-AI collaboration, containing 400 sentences from News, WikiNews, and Wikipedia genres. Each sentence has an average of four complex words, and each complex word has an average of five simplified substitutes.
  • Evaluation Datasets: The newly constructed dataset is used for evaluating the proposed methods.

Key Insights and Ideas

  • LLMs Disrupt the LS Pipeline: LLMs can effectively perform LS in a single step, making the traditional multi-stage pipeline obsolete for evaluating these models.
  • Existing Datasets are Inadequate: Current LS datasets are not designed to evaluate end-to-end sentence simplification by LLMs.
  • Human-AI Collaboration is Key: Building a comprehensive LS dataset requires combining the efficiency of LLMs with human judgment for quality and nuance.
  • Multi-LLM Synergy: A collaborative approach where multiple LLMs work together, mimicking expert teams, significantly boosts performance beyond single-LLM or traditional methods.
  • Validation Step is Crucial: A sentence-level validation step, assessing simplicity and faithfulness, is more effective than just ranking individual word substitutes.

Key Results

  • Multi-LLM Collaboration Outperforms: The proposed CoLLS method, using multiple LLMs, significantly outperforms single-prompt LLMs (including GPT3.5 and Llama3 with COT) and a strong baseline (LSBert) across various datasets.
  • High Quality Dataset: The newly constructed dataset demonstrates high precision (over 94%) and coverage (95%) for substitutes, indicating its quality.
  • LLM Usefulness in Annotation: LLMs can effectively assist human annotators, reducing workload and improving efficiency.
  • CoLLS Performance: For example, on the WikiNews dataset, COLLS(GPT3.5) achieved an F1 score of 0.605 and F1-20 of 0.247, outperforming other methods.

Limitations and Future Work

  • Dataset Genre Limitation: The current dataset is limited to News and Wiki genres, potentially affecting applicability in other contexts.
  • Dataset Size: The dataset comprises 400 instances, which may not fully capture linguistic diversity.
  • Future Work: The authors plan to expand the dataset to include more text types and languages. They also aim to further explore the effectiveness of multi-LLM collaboration for LS.

Summary updated

Abstract

Lexical Simplification (LS) methods use a three-step pipeline: complex word identification, substitute generation, and substitute ranking, each with separate evaluation datasets. We found large language models (LLMs) can simplify sentences directly with a single prompt, bypassing the traditional pipeline. However, existing LS datasets are not suitable for evaluating these LLM-generated simplified sentences, as they focus on providing substitutes for single complex words without identifying all complex words in a sentence. To address this gap, we propose a new annotation method for constructing an all-in-one LS dataset through human-machine collaboration. Automated methods generate a pool of potential substitutes, which human annotators then assess, suggesting additional alternatives as needed. Additionally, we explore LLM-based methods with single prompts, in-context learning, and chain-of-thought techniques. We introduce a multi-LLMs collaboration approach to simulate each step of the LS task. Experimental results demonstrate that LS based on multi-LLMs approaches significantly outperforms existing baselines.
BibTeX
@misc{qiang2025new,
  author = {Jipeng Qiang and Minjiang Huang and Yi Zhu and Yunhao Yuan and Chaowei Zhang and Xiaoye Ouyang},
  title = {{New Evaluation Paradigm for Lexical Simplification}},
  year = {2025},
  eprint = {2501.15268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.15268},
}
Figure