Beyond Overlap Metrics for Text Simplification

This direction is merged into another direction: simplification evaluation gap. Its last version stays readable and citable here.

Version 1 of 113 papers18 conceptsfrozen 2026-10-04

In most natural language processing tasks, success is measured by how closely a model's output matches a human reference. In Machine Translation (MT), metrics like BLEU reward models for high n-gram overlap with the target. However, in Text Simplification (TS), this "similarity-to-reference" logic breaks down. Because simplification inherently requires transforming the source text—through lexical substitution, structural reordering, or content deletion—standard metrics often end up rewarding models for doing nothing at all or, conversely, for deleting too much.

The Paradox of Similarity

The fundamental problem with using MT metrics for simplification is that they prioritize lexical similarity to the source. As Zhemin Zhu and colleagues argued in A Monolingual Tree-based Translation Model for Sentence Simplification, effective simplification necessitates significant deviations from the original sentence structure. When a model successfully simplifies a sentence, it often reduces the n-gram overlap with the complex source.

Standard metrics like BLEU and ROUGE inadvertently penalize these successful transformations. This creates a perverse incentive: a system that performs no simplification and simply copies the complex source text can achieve a higher score than a system that successfully reduces complexity but introduces new, simpler vocabulary.

The "Do Nothing" Trap

This evaluation gap is not just theoretical. In A Deeper Exploration of the Standard PB-SMT Approach to Text Simplification and its Evaluation, Sanja Štajner and her team demonstrated through 40 controlled experiments that BLEU is a poor proxy for simplification quality. They found that the metric frequently rewards systems for "doing nothing," as preserving the original text maintains high n-gram overlap.

This issue is particularly acute in high-stakes domains like medicine. Zifei Zheng et al. noted in Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation that standard metrics fail to distinguish between a model that preserves dense clinical jargon (like "NSTEMI") and one that successfully translates it into lay language. Similarly, Zihao Li et al. found in Large Language Models for Biomedical Text Simplification: Promising But Not There Yet that high ROUGE scores often indicate a failure to simplify, as the model has likely just copied the complex biomedical source text verbatim.

The Deletion Bias

If overlap metrics reward copying, they also suffer from a second flaw: they can be "gamed" by excessive deletion. A model can achieve high precision by simply removing any complex phrases it doesn't understand, rather than attempting to simplify them.

Mounica Maddela and Wei Xu highlighted this in Controllable Text Simplification with Explicit Paraphrasing, noting that models often simplify primarily through content removal to boost scores. This problem scales with the length of the text. In Document-Level Text Simplification: Dataset, Criteria and Baseline, Renliang Sun and Xiaojun Wan observed that metrics may reward extreme brevity, allowing models to discard the majority of a document's information while still appearing "successful" according to automated benchmarks.

Towards Readability and Transformation

To move past these limitations, researchers are shifting focus toward metrics that explicitly measure transformation and readability.

  1. Readability Metrics: Yue Guo and Trevor Cohen emphasize in Automated Lay Language Summarization of Biomedical Scientific Reviews the need for metrics that specifically track the translation of technical jargon into plain language, rather than just n-gram overlap.
  2. Transformation-Aware Metrics: New benchmarks like D-SARI, introduced by Renliang Sun et al., attempt to account for the complexities of document-level rewriting, where simple deletion is insufficient for maintaining cohesion.
  3. Controllable Generation: By moving away from "black box" models and toward systems that allow explicit tuning of splitting, deletion, and paraphrasing—as seen in the work of Mounica Maddela—researchers can better evaluate whether a model is truly simplifying or just truncating.

The consensus across these studies is clear: for text simplification to advance, the field must stop treating it as a standard translation task. Success should be measured not by how much of the source remains, but by how effectively the complexity has been transformed.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_beyond_overlap_metrics_for_text_simplifi_6d7832c8,
  title = {Beyond Overlap Metrics for Text Simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/beyond-overlap-metrics-for-text-simplification/rdr_6d7832c8a71945cd9fc5cc67d2c559c5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_beyond_overlap_metrics_for_text_simplifi_6d7832c8_v1,
  title = {Beyond Overlap Metrics for Text Simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/beyond-overlap-metrics-for-text-simplification/rdr_6d7832c8a71945cd9fc5cc67d2c559c5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}