Evaluating Text Simplification Beyond Lexical Overlap

214 concepts111 papers1 proposal

In machine translation or text summarization, standard metrics like BLEU and ROUGE measure success by calculating n-gram overlap between a model's output and a reference text. If the output closely matches the reference, the score is high.

However, when applied to text simplification, this paradigm creates a fundamental paradox. The primary goal of text simplification is to transform text—by replacing complex jargon, splitting sentences, reordering clauses, or deleting redundant information. Consequently, metrics that reward high lexical overlap with the source text inherently penalize the very operations required to make text more accessible. As Zhu et al. and Štajner et al. demonstrate, this creates a "do-nothing" bias: a model that performs absolutely no simplification and simply copies the complex input can achieve a higher BLEU score than a model that successfully simplifies the text.

The Failure Modes of Overlap Metrics

The misalignment between automated metrics and human judgment manifests in several distinct failure modes across different types of text transformations.

Penalizing Lexical and Structural Deviations

High-quality plain language summaries must deviate significantly from the technical vocabulary of the source text to be understandable to lay audiences. Because metrics like ROUGE and METEOR prioritize lexical similarity, they interpret these intentional vocabulary shifts as a loss of information or accuracy (Guo et al.; Yao et al.). Similarly, for structural tasks like "Split and Rephrase"—where long, complex sentences are broken into shorter ones—traditional overlap metrics penalize the necessary rephrasing and structural changes (Kim et al.).

As Sulem argues, these metrics function primarily as proxies for the degree of textual change rather than measures of output quality, showing a conservative bias that rewards models for staying close to the original text. Furthermore, Alva-Manchego demonstrates that automated metrics exhibit a severe lack of correlation with human simplicity ratings when text undergoes multiple simultaneous abstractive transformations, such as combining splitting, compression, and reordering.

Blindness to Factuality and Hallucinations

A critical flaw of standard simplification metrics is their inability to evaluate semantic integrity. In high-stakes domains like medicine, text simplification models frequently introduce factual errors, omissions, or hallucinations. Devaraj demonstrates that standard metrics like SARI show almost no correlation with the factual accuracy of generated outputs. A model can be optimized for high lexical change and readability scores while simultaneously introducing severe factual distortions that metrics like SARI fail to penalize (Devaraj; Joseph et al.).

The Deletion and Brevity Trap

At the document level, standard metrics often fail because they do not adequately penalize models that achieve high scores through extreme brevity or the wholesale deletion of complex content (Sun et al.). Without length and sentence-count penalties, a model can appear highly successful by simplifying only a tiny fraction of the text and discarding the rest, failing to preserve the core information of the original document.

The Readability vs. Comprehension Mismatch

Another layer of the evaluation gap is the discrepancy between algorithmic readability formulas (such as Flesch Reading Ease) and actual human comprehension.

Automated metrics often exhibit a strong bias toward brevity and low reading levels, which can lead to oversimplification. However, human readers frequently prefer text that retains intermediate definitions or explanations of technical terms. Fankhauser shows that automated selection systems optimized for extreme readability scores select outputs that human judges find less helpful than more detailed, explanatory text.

Furthermore, standard metrics are completely blind to elaborative simplification—the addition of explanatory context or definitions not present in the source text. As Maddela notes, existing sentence-level metrics cannot accurately assess the quality or relevance of this newly introduced information. This creates a scenario where models with lower automated scores can actually achieve higher human ratings for readability and informativeness (Hou et al.).

Designing Better Evaluators

To close the metric-human alignment gap, researchers are shifting toward evaluation frameworks that capture the multi-dimensional nature of simplification.

Reference-Free and Absolute Simplicity

Traditional metrics focus exclusively on the difference between the source and the simplified text. To address this, Kreutz et al. introduced BATS, a reference-free method that leverages linguistic rules to assess the absolute simplicity of a text, better aligning with how accessible the text is to a general audience.

Learnable Metrics

Instead of relying on rigid n-gram matching, recent approaches train supervised models directly on human qualitative judgments. Maddela et al. developed LENS, a learnable evaluation metric trained on diverse human ratings. LENS achieves more than double the correlation with human intuition compared to traditional metrics like SARI or BERTScore because it can model the complex trade-offs between paraphrasing, deletion, and sentence splitting.

Systematic Meta-Evaluation

To ensure that metrics themselves are robust, frameworks like APPLS (Guo et al.) use targeted perturbations to stress-test evaluation tools. By breaking summaries with specific linguistic errors, researchers can quantify exactly where metrics fail, paving the way for evaluation frameworks that truly reflect human comprehension and communicative effectiveness.

Key Foundational Work

A Monolingual Tree-based Translation Model for Sentence Simplification (2010) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_simplification_evaluation_gap_c5db712d,
  title = {simplification evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/simplification-evaluation-gap/rdr_c5db712dddb7456d9595e741911f3103},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_simplification_evaluation_gap_c5db712d_v1,
  title = {simplification evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/simplification-evaluation-gap/rdr_c5db712dddb7456d9595e741911f3103/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake