TL;DR
Most training-free methods for shortening LLM responses are actually just "truncation in disguise," performing no better than simply cutting an answer short. This paper introduces ELIDE, a novel decoding-time mechanism that hides redundant sentences from the reader while maintaining the model's internal state, successfully beating the truncation baseline—though it reveals that automatic metrics are dangerously blind to the subtle ways these edits can harm an answer.
What Problem Does This Paper Address?
Large Language Models (LLMs) are notoriously wordy. Because reward models and human judges often correlate "longer" with "better," instruction-tuned models are systematically pushed toward verbosity. This isn't just annoying; it’s expensive. Every extra token requires a full forward pass, consuming time, energy, and the reader's attention.
The authors set out to find a training-free way to cut this surplus length without losing content. Crucially, they establish a rigorous evaluation bar: the truncation frontier. They argue that any method claiming to reduce verbosity must prove it retains more information than simply truncating the original, unmodified answer to the same length. If a method can't beat simple truncation, it hasn't reduced verbosity; it has just shortened the text.
Methods
The researchers investigated two main approaches:
- Entropy-Gated Logits Processor (EGP): A controller that acts inside the decoding loop to encourage the model to stop early when the "entropy" (uncertainty) of the next token distribution is high. While it successfully shortened answers, the authors mathematically proved—and confirmed via audit—that EGP is essentially a sophisticated stop-controller. Its output is always just a prefix of the original answer, meaning it never actually "selects" better content than simple truncation.
- Entailment-Licensed In-Decoding Elision (ELIDE): This is the paper's primary contribution. Instead of changing what the model generates, ELIDE changes what the reader sees. As the model generates text, ELIDE segments it into spans (usually sentences). It then uses a separate Natural Language Inference (NLI) model to check if the new span is already "entailed" (implied) by the text already shown.
- If a sentence is redundant, it is elided (hidden from the reader).
- Crucially, the hidden text remains in the model's KV cache. This ensures "trajectory invariance"—the model's internal state doesn't drift, and it continues to generate exactly what it would have otherwise.
Datasets
The authors evaluated their methods across several popular benchmarks and model families:
- Models: Mistral-7B, Llama-3.1-8B, and Qwen2.5-3B, with transfer tests on OLMo-2 and Granite-3.1.
- Benchmarks: MT-Bench (dev and sealed test splits), AlpacaEval, Arena-Hard-Auto, and YapBench.
- Custom Data: They also used hand-written multi-part prompts for initial screening and two "fresh" test frames to validate their final version (v2).
Key Insights and Ideas
- The Truncation Trap: A survey of 23 training-free mechanism families found that almost none could beat the truncation frontier. Many methods that seemed to work were actually "drifting" (answering a simpler version of the question) or "rerouting" (substituting different content).
- LLMs are Blind to Their Own Redundancy: The authors found a fascinating "diagnostic" gap: while LLMs can perfectly follow a redundancy signal if given one by an oracle, they are at-chance (AUROC 0.495) at identifying which of their own sentences are redundant. They can act on the signal, but they cannot compute it.
- The Failure of Automatic Metrics: This is perhaps the most critical finding. Automatic measures like BERTScore and NLI coverage suggested ELIDE v1 was performing perfectly. However, a manual reading audit revealed that 59% of its elisions actually harmed the answer (e.g., deleting the actual answer because it looked like a restatement of the question).
- Discourse Scoping is Key: The harm in ELIDE v1 often came from the NLI model being too good—it would see that a sentence in a list item was entailed by a sentence in a different list item and delete it. ELIDE v2 fixes this by restricting the "license" to the current discourse unit (e.g., only eliding a sentence if it's redundant within the same list item).
Key Results
- EGP reduced tokens by 25–35%, but as a stop-controller, it sits exactly on the truncation frontier.
- ELIDE v1 successfully beat the truncation frontier on three model families, but at the cost of high harm rates (up to 65% in some tests).
- ELIDE v2 (Scoped) significantly improved safety. It reduced the harmful share of elisions to roughly 1 in 10 while still beating the truncation baseline.
- Token Savings: The actual amount of restatement in typical LLM answers is small (2–6%). Consequently, ELIDE v2 provides modest but "pure" verbosity reduction, typically around 1% of shown tokens, which increases as answers get longer.
Limitations and Future Work
The authors acknowledge that the savings from ELIDE are currently small because instruction-tuned models don't actually repeat themselves that much—their "excess" is often just unnecessary detail rather than literal restatement.
Furthermore, even with the scoped version, a 10% harm rate is significant for production systems. The authors suggest that future work should focus on deriving the elision "license" from more sophisticated discourse structures and testing the method on long-form tasks like retrieval-augmented generation (RAG), where redundancy is likely much higher. Finally, they emphasize that safety claims for any text-omission method must rely on human reading, as current automatic judges are not yet reliable enough to catch semantic damage.
Summary updated