Beyond Truncation? Entailment-Licensed Elision and the Limits of Semantic Preservation in LLM Decoding

Mahmoud Alyosify, Mirna Imbabi, Eman ElkhamisiarXiv 2026 · arXiv

TL;DR

Most training-free methods for shortening LLM responses are actually just "truncation in disguise," performing no better than simply cutting an answer short. This paper introduces ELIDE, a novel decoding-time mechanism that hides redundant sentences from the reader while maintaining the model's internal state, successfully beating the truncation baseline—though it reveals that automatic metrics are dangerously blind to the subtle ways these edits can harm an answer.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notoriously wordy. Because reward models and human judges often correlate "longer" with "better," instruction-tuned models are systematically pushed toward verbosity. This isn't just annoying; it’s expensive. Every extra token requires a full forward pass, consuming time, energy, and the reader's attention.

The authors set out to find a training-free way to cut this surplus length without losing content. Crucially, they establish a rigorous evaluation bar: the truncation frontier. They argue that any method claiming to reduce verbosity must prove it retains more information than simply truncating the original, unmodified answer to the same length. If a method can't beat simple truncation, it hasn't reduced verbosity; it has just shortened the text.

Methods

The researchers investigated two main approaches:

  1. Entropy-Gated Logits Processor (EGP): A controller that acts inside the decoding loop to encourage the model to stop early when the "entropy" (uncertainty) of the next token distribution is high. While it successfully shortened answers, the authors mathematically proved—and confirmed via audit—that EGP is essentially a sophisticated stop-controller. Its output is always just a prefix of the original answer, meaning it never actually "selects" better content than simple truncation.
  2. Entailment-Licensed In-Decoding Elision (ELIDE): This is the paper's primary contribution. Instead of changing what the model generates, ELIDE changes what the reader sees. As the model generates text, ELIDE segments it into spans (usually sentences). It then uses a separate Natural Language Inference (NLI) model to check if the new span is already "entailed" (implied) by the text already shown.
  3. If a sentence is redundant, it is elided (hidden from the reader).
  4. Crucially, the hidden text remains in the model's KV cache. This ensures "trajectory invariance"—the model's internal state doesn't drift, and it continues to generate exactly what it would have otherwise.
Figure 1: ELIDE. Each completed span passes the safeguards, the cosine pre-filter and the NLI licence; a licensed span is withheld from the reader but stays in the KV cache, so the model’s generation is unchanged. The processor’s length term can optionally be attached as a logits hook.

Datasets

The authors evaluated their methods across several popular benchmarks and model families:

  • Models: Mistral-7B, Llama-3.1-8B, and Qwen2.5-3B, with transfer tests on OLMo-2 and Granite-3.1.
  • Benchmarks: MT-Bench (dev and sealed test splits), AlpacaEval, Arena-Hard-Auto, and YapBench.
  • Custom Data: They also used hand-written multi-part prompts for initial screening and two "fresh" test frames to validate their final version (v2).

Key Insights and Ideas

  • The Truncation Trap: A survey of 23 training-free mechanism families found that almost none could beat the truncation frontier. Many methods that seemed to work were actually "drifting" (answering a simpler version of the question) or "rerouting" (substituting different content).
  • LLMs are Blind to Their Own Redundancy: The authors found a fascinating "diagnostic" gap: while LLMs can perfectly follow a redundancy signal if given one by an oracle, they are at-chance (AUROC 0.495) at identifying which of their own sentences are redundant. They can act on the signal, but they cannot compute it.
  • The Failure of Automatic Metrics: This is perhaps the most critical finding. Automatic measures like BERTScore and NLI coverage suggested ELIDE v1 was performing perfectly. However, a manual reading audit revealed that 59% of its elisions actually harmed the answer (e.g., deleting the actual answer because it looked like a restatement of the question).
  • Discourse Scoping is Key: The harm in ELIDE v1 often came from the NLI model being too good—it would see that a sentence in a list item was entailed by a sentence in a different list item and delete it. ELIDE v2 fixes this by restricting the "license" to the current discourse unit (e.g., only eliding a sentence if it's redundant within the same list item).

Key Results

  • EGP reduced tokens by 25–35%, but as a stop-controller, it sits exactly on the truncation frontier.
  • ELIDE v1 successfully beat the truncation frontier on three model families, but at the cost of high harm rates (up to 65% in some tests).
  • ELIDE v2 (Scoped) significantly improved safety. It reduced the harmful share of elisions to roughly 1 in 10 while still beating the truncation baseline.
  • Token Savings: The actual amount of restatement in typical LLM answers is small (2–6%). Consequently, ELIDE v2 provides modest but "pure" verbosity reduction, typically around 1% of shown tokens, which increases as answers get longer.
Figure 2: The truncation frontier on Qwen2.5-3B (MT-Bench development): (a) coverage recall of the baseline and (b) entity retention against token reduction. The processor’s length term sits on the frontier, as Proposition 1 requires; contrastive verbosity decoding, which edits token choices, falls far below it.

Limitations and Future Work

The authors acknowledge that the savings from ELIDE are currently small because instruction-tuned models don't actually repeat themselves that much—their "excess" is often just unnecessary detail rather than literal restatement.

Furthermore, even with the scoped version, a 10% harm rate is significant for production systems. The authors suggest that future work should focus on deriving the elision "license" from more sophisticated discourse structures and testing the method on long-form tasks like retrieval-augmented generation (RAG), where redundancy is likely much higher. Finally, they emphasize that safety claims for any text-omission method must rely on human reading, as current automatic judges are not yet reliable enough to catch semantic damage.

Summary updated

Abstract

Instruction-tuned large language models (LLMs) answer at greater length than a request needs. We ask whether a training-free controller inside the decoding loop can remove that surplus while preserving content, judged against the simplest possible baseline: truncating the model's own answer to the same length. An entropy-gated logits processor removes 25.5-34.9% of tokens on MT-Bench development questions and 19.9-34.2% on a sealed test split within pre-registered BERTScore non-inferiority margins, yet we prove and confirm by audit that its output is always a prefix of the unmodified answer: the saving is truncation. A survey of 23 training-free mechanism families finds none that selects content better than truncation without oracle labels. We therefore introduce Entailment-Licensed In-Decoding Elision (ELIDE), which, at each sentence boundary, withholds a span from the reader only when a natural-language inference model judges it entailed by the text already shown, while leaving the model's generation unchanged. ELIDE beats matched-length truncation on three model families, but reading 936 of its elisions on 670 public prompts shows that 59% harm the answer, a failure that no automatic measure detects. Scoping the licence to its discourse unit reduces the harmful share to about one elision in ten on two fresh tests.
BibTeX
@misc{alyosify2026beyond,
  author = {Mahmoud Alyosify and Mirna Imbabi and Eman Elkhamisi},
  title = {{Beyond Truncation? Entailment-Licensed Elision and the Limits of Semantic Preservation in LLM Decoding}},
  year = {2026},
  eprint = {2611.08975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08975},
}
Figure