The Failure of Lexical Overlap in Text Evaluation

140 concepts129 papers3 proposals

For decades, the standard recipe for evaluating Natural Language Generation (NLG) has relied on string matching. Metrics like BLEU and ROUGE, originally popularized by researchers such as Chin-Yew Lin in Automated Text Summarization, calculate the surface-level overlap of n-grams between a machine-generated output and a human-written reference.

If the words match, the score is high. If they do not, the score is low.

While this approach was computationally cheap and sufficient for early translation and summarization systems, the rise of Large Language Models (LLMs) has exposed a fundamental flaw: lexical similarity does not equal semantic alignment. Traditional metrics evaluate the superficial structure of text rather than its underlying meaning, creating a severe "semantic evaluation gap" that fails to reflect true human judgment.

The Mechanics of Lexical Failure

To understand why lexical overlap fails, we must look at how language maps meaning to surface text. A single semantic concept can be expressed through thousands of distinct lexical combinations, while a minor change in word order can completely invert the truth value of a sentence. N-gram metrics operate on the blind assumption that text distance correlates linearly with semantic distance. This assumption breaks down in two distinct ways: the Paraphrase Penalty and the Keyword Illusion.

The Paraphrase Penalty (False Negatives)

When an advanced generative model produces a creative, fluent, and entirely accurate response using synonyms or alternative syntactic structures, lexical metrics penalize it heavily.

As noted by David Chan et al. in CLAIR: Evaluating Image Captions with Large Language Models, factually accurate descriptions that rely on paraphrasing receive low scores from traditional metrics, despite being highly rated by humans. Ramya Akula et al. demonstrate in Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization that exact word matching cannot recognize semantic equivalence across diverse phrasing. For example, the sentences "The projectile struck the vehicle" and "The rocket hit the truck" share zero non-filler n-grams, resulting in a near-zero ROUGE score despite conveying identical information.

The Keyword Illusion (False Positives)

Conversely, a model can generate a response that achieves a high lexical overlap score while being completely incoherent or factually wrong. Because n-gram metrics treat tokens with relatively uniform structural weight, they are easily gamed by models that output the correct keywords in an incorrect logical order.

Mohit Bansal et al. argue in Multi-Reward Reinforced Summarization with Saliency and Entailment that standard metrics fail to distinguish between critical information-bearing entities and superficial filler words. This encourages models to prioritize keyword-stuffing over capturing core meaning.

In fact, a model can corrupt critical entities like dates, negations, or locations and still maintain an excellent overlap score. As Vinay Rao demonstrates in Assessing The Factual Accuracy of Generated Text, n-gram metrics routinely assign high scores to summaries containing catastrophic factual errors because they prioritize surface-level vocabulary presence over semantic truth.

Reference: "The patient should take the medicine, not avoid it."
Candidate: "The patient should avoid the medicine, not take it."

Lexical Overlap: 100% (Perfect Match)
Semantic Alignment: 0% (Total Contradiction)

The Structural Blindspot in Complex Tasks

The disconnect between lexical overlap and human judgment becomes even more pronounced in long-form generation, structured data synthesis, and reasoning tasks.

In long-form question answering, domain experts prioritize completeness and factual accuracy over the superficial conciseness that n-gram metrics favor. Fangyuan Xu et al. show in A Critical Evaluation of Evaluations for Long-form Question Answering that no traditional automatic metric reliably captures overall answer quality in extended text. Similarly, Kalpesh Krishna et al. reveal in Hurdles to Progress in Long-form Question Answering that automated metrics focusing on surface similarity can be easily gamed by models producing fluent but entirely ungrounded text, masking a lack of real-world utility.

When synthesizing structured data, such as reasoning over tables, the failure is absolute. Yuqi Wang et al. demonstrate in Revisiting Automated Evaluation for Long-form Table Question Answering that BLEU and ROUGE exhibit near-zero correlation with human judgments because they reward text that shares words but contains logical hallucinations derived from the data. Furthermore, as Caiqi Zhang et al. note in LUQ: Long-text Uncertainty Quantification for LLMs, high lexical similarity between multiple model outputs can actively mask significant factual contradictions in long-form content, making surface-level consistency a dangerous proxy for certainty or truth.

The Shift to Model-Based Evaluation

To bridge the semantic evaluation gap, the NLP community is shifting away from string matching toward model-based evaluation paradigms that analyze text within a continuous semantic space.

[Surface Text] ───(Traditional Metrics)───> Brittle N-Gram Counting
       │
 (Model Embedding)
       ▼
[Semantic Space] ───(Modern Evaluators)───> Factual & Logical Alignment

This evolution has progressed through several distinct strategies:

  1. Question-Answering Verification: Instead of comparing strings, metrics like those proposed by Matan Eyal et al. in Question Answering as an Automatic Evaluation Metric for News Article Summarization and Nan Feng in Improving Factual Consistency of Abstractive Summarization via Question Answering generate questions from the reference text and check if the candidate text can correctly answer them. This explicitly measures information recall rather than vocabulary matching.
  2. Multi-Dimensional Unified Evaluators: Frameworks such as UniEval, developed by Heng Ji et al. in Towards a Unified Multi-Dimensional Evaluator for Text Generation, frame evaluation as a boolean question-answering task to independently score distinct dimensions like coherence, naturalness, and factual consistency.
  3. LLM-as-a-Judge: Modern frameworks leverage the instruction-following capabilities of frontier models to evaluate text quality directly. For example, G-Eval, introduced by Yang Liu et al. in G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment, combines Chain-of-Thought prompting with probability-weighted scoring to achieve significantly higher alignment with human perceptions of creativity and quality than any n-gram metric could provide.

By moving from literal string matching to semantic and logical verification, these modern approaches ensure that text generation is evaluated on the substance of what is said, rather than merely the specific words chosen to say it.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lexical_overlap_limitations_n_gram_overl_f00529e9,
  title = {lexical overlap limitations (n-gram overlap limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-overlap-limitations-n-gram-overlap-limitations/rdr_f00529e95cd14df48843471e48ae48d0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lexical_overlap_limitations_n_gram_overl_f00529e9_v1,
  title = {lexical overlap limitations (n-gram overlap limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-overlap-limitations-n-gram-overlap-limitations/rdr_f00529e95cd14df48843471e48ae48d0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake