Bridging the Comprehension Gap in Automated Text Simplification

This direction is merged into another direction: simplification evaluation gap. Its last version stays readable and citable here.

Version 1 of 143 papers56 conceptsfrozen 2026-10-04

In the field of Natural Language Processing, we often assume that if a model achieves a high score on a standard benchmark, it has mastered the task. However, in text simplification—the art of making complex information accessible to non-experts or low-literacy readers—this assumption is increasingly under fire. A growing body of research suggests a profound "alignment gap" between what our algorithms optimize for and what human readers actually understand.

The Problem of Metric Laziness

Most text simplification models are evaluated using metrics like BLEU, ROUGE, or SARI. These metrics primarily measure "overlap"—how many words the model's output shares with a human-written reference.

The danger of this approach is what Xingxing Zhang and Mirella Lapata call "copy-paste laziness." Because these metrics penalize structural changes and word deletions, models often learn to simply repeat the source text with minor tweaks. As Renliang Sun and Xiaojun Wan observe, this creates a misalignment where models that perform the least amount of actual simplification receive the highest scores.

The Readability Paradox

Even when we use specialized readability formulas like Flesch-Kincaid, we often miss the mark. These formulas typically rely on surface-level features like sentence length and syllable counts.

Isabel Cachola and Mark Dredze have shown that these metrics correlate poorly with actual human judgment. In fact, in high-stakes domains like law or medicine, the relationship can even reverse. Lekang Jiang identifies a "readability paradox" in patent law: successful revisions often increase technical complexity to ensure legal robustness, a nuance that standard metrics penalize as "bad" simplification.

Similarly, Pascal Severin Andermatt and Tobias Fankhauser found that while automated tools prefer "5-year-old" level explanations, human readers actually prefer more detailed versions that include definitions of technical terms. A text that is "too simple" can lose the very information the reader needs to learn.

Perceived Simplicity vs. Actual Comprehension

Perhaps the most sobering discovery in recent work is the "comprehension gap." Yue Guo and Trevor Cohen conducted large-scale studies showing that readers often feel they understand an AI-generated summary (subjective rating), yet fail objective tests about its content (factual recall).

This suggests that our current evaluation frameworks are measuring "fluency" (how smooth the text sounds) rather than "communicative effectiveness." As Sebastian Joseph and Wei Xu note in their work on medical texts, a model can be highly fluent while hallucinating dangerous factual errors, a trade-off that standard metrics like BLEU are blind to.

Toward Learnable and Human-Centric Metrics

To fix this, the field is moving toward two new frontiers:

  1. Learnable Metrics: Instead of counting word overlaps, researchers like Mounica Maddela and Wei Xu have developed LENS. This is a model trained specifically on human judgments to recognize the complex trade-offs between sentence splitting, paraphrasing, and deletion.
  2. Targeted Simplification: Rather than rewriting entire documents, researchers like Sumit Asthana and Elizabeth Clark are focusing on "targeted concept simplification"—explaining difficult terms in context. This approach prioritizes the reader's educational utility over statistical similarity to a reference.

The consensus across these 39 papers is clear: making text "simple" is not a math problem of word counts; it is a cognitive problem of information transfer. As Sandra M. Aluísio argued early on in the PorSimples project, the only true measure of a simplification system is whether the target audience can actually use the information to make decisions in their lives.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bridging_the_comprehension_gap_in_automa_bf9c345c,
  title = {Bridging the Comprehension Gap in Automated Text Simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-comprehension-gap-in-automated-text-simplification/rdr_bf9c345c9a6946219949ce99ae909a95},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_comprehension_gap_in_automa_bf9c345c_v1,
  title = {Bridging the Comprehension Gap in Automated Text Simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-comprehension-gap-in-automated-text-simplification/rdr_bf9c345c9a6946219949ce99ae909a95/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}