For years, the field of text summarization relied on a simple assumption: a good summary is one that shares many words with a human-authored reference. This assumption is baked into lexical overlap metrics like ROUGE, which count matching $n$-grams to calculate a quality score. While this approach worked reasonably well for extractive summarization—where a model simply selects and pastes sentences verbatim from the source text—it breaks down completely when applied to abstractive summarization.
Abstractive summarizers behave more like humans: they comprehend a document and rewrite it using novel vocabulary, paraphrasing, and structural synthesis. Because lexical overlap metrics only look at surface-level word matches, they create a fundamental misalignment between automated scores and actual summary quality.
The Paraphrasing Penalty
The most immediate failure of $n$-gram metrics is that they actively penalize the very creativity that defines abstractive summarization. If a model generates an elegant, highly condensed paraphrase using synonyms, an $n$-gram metric will score it poorly simply because the exact words do not match the reference text.
As Hailu et al. observe, abstractive summaries convey meaning through entirely new sentence structures and word choices, making direct lexical matching an ill-suited evaluation strategy. This mismatch creates what Kryscinski et al. describe as a structural tension in model training: when researchers optimize a model to improve its creative abstraction, its scores on traditional benchmarks often drop.
Source Text: "The celestial body illuminated the nocturnal landscape."
Human Reference: "The moon shone at night."
Extractive Model: "The celestial body illuminated the landscape."
--> High Lexical Overlap (ROUGE), Low Abstraction
Abstractive Model: "The lunar light brightened the dark evening."
--> Zero Lexical Overlap (ROUGE), High Quality Paraphrase
This gap has widened with the advent of Large Language Models (LLMs). In a systematic survey, Haopeng Zhang et al. point out that modern abstractive models frequently generate novel phrasing that human evaluators prefer over the original human reference texts. Yet, because these summaries lack literal word matching, traditional metrics assign them low scores. Furthermore, Ng et al. show that $n$-gram metrics penalize models for removing redundant words that happen to be present in human references, rewarding wordiness over concise, efficient synthesis.
Blindness to Logic, Flow, and Truth
Beyond penalizing vocabulary diversity, lexical metrics are fundamentally blind to the semantic and structural properties that make text readable and accurate. An $n$-gram metric evaluates a text as a "bag of words" or short phrases, ignoring how those phrases connect.
1. Narrative Flow and Coherence
A summary can achieve a high ROUGE score by stringing together key phrases from the reference text, while remaining completely unreadable. Cardenas et al. demonstrate that summaries designed to maximize lexical overlap often lack narrative flow, resulting in a disjointed reading experience that humans find difficult to follow. Similarly, Deutsch et al. emphasize that word-level similarity cannot measure the logical coherence between a generated sentence and its preceding context.
2. Factual Consistency and Hallucinations
Lexical overlap is a poor proxy for truth. A model can change a single word—such as turning "violently rejected" into "warmly accepted"—and maintain a 90% lexical overlap score while completely reversing the meaning of the text. Pham et al. note that high scores in lexical overlap offer no guarantee against hallucinations or factual distortions. To address this, Feng et al. argue that the field must transition to evaluation frameworks that verify factual alignment in rewritten text without relying on surface-level word matches.
3. Nuance, Sentiment, and Constraints
When summaries must capture specific dimensions like user constraints or emotional tone, lexical metrics fail entirely:
- Controllable Summarization: If a user asks for a summary focused on a specific aspect of a document, the output will naturally deviate from a generic human reference. Wang et al. show that word-overlap metrics fail to account for this utility, penalizing customized summaries for doing exactly what they were designed to do.
- Sentiment Preservation: In opinion summarization, Jayanth et al. explain that word-level metrics cannot verify whether a condensed text maintains the emotional polarity and intensity of the source.
- Argumentative Structure: For dialogue and meeting summaries, Barker et al. highlight that $n$-gram metrics cannot assess whether a summary accurately reflects the balance of opinions or the logical flow of a debate.
The Extractive Bias of Evaluation Frameworks
Why did these inadequate metrics persist for so long? The answer lies in an architectural bias: many evaluation frameworks and meta-evaluation datasets were originally validated on purely extractive data.
When a metric is optimized for extractive tasks, it learns to look for specific surface features. However, Pitler et al. reveal that features indicating high linguistic quality in synthesized text can signal poor quality in extractive contexts. For example, discourse connectives (like "however" or "therefore") make abstractive text flow smoothly, but in extractive summaries, they often point to missing context because the surrounding sentences were deleted.
Consequently, metrics validated on extractive datasets lack the sensitivity required to assess abstractive systems, which exhibit entirely different grammatical and stylistic profiles (Vadlapudi et al.). As ShafieiBavani et al. warn, when the lexical overlap between a summary and a reference is minimal, traditional metrics lose their evaluative power.
Even newer, "reference-free" evaluation metrics—which score a summary by comparing it directly to the source text instead of a human reference—fall into this trap. Durmus et al. discovered that these metrics often rely on spurious shortcuts like text length and word density. When forced to evaluate high-quality abstractive systems that use novel phrasing, these reference-free metrics lose their primary signal and perform worse than simple lexical baselines.
Moving Beyond N-Grams
To break free from the limitations of lexical overlap, the research community is shifting toward metrics that evaluate meaning rather than words.
In an analysis of long-form narrative summarization, Kashyap demonstrates that semantic similarity metrics, such as BERTScore, are significantly better suited for evaluating heavily rephrased content than ROUGE. Similarly, for complex tasks like timeline summarization, Chenlong Zhang et al. emphasize that metrics must evolve to measure factual nuances and structural consistency across different levels of detail, rather than counting shared $n$-grams.
Ultimately, as abstractive models become more sophisticated, our evaluation metrics must follow suit—moving away from counting identical characters and toward understanding shared ideas.
Key Foundational Work
Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality (2008) — contributed 1 concept elements to this direction.
A Methodology for Extrinsic Evaluation of Text Summarization: Does ROUGE Correlate? (2005) — contributed 1 concept elements to this direction.
Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization (2005) — contributed 1 concept elements to this direction.