Say Less, Mean Better: Training-Free Verbosity Control for Language Models

Mahmoud Alyosify, Mirna Imbabi, Eman Elkhamisi, Rahatara FerdousiarXiv 2026 · arXiv

TL;DR

MinimalLM is a training-free framework that slashes Large Language Model (LLM) verbosity by nearly 50% while ensuring responses remain structurally complete. By combining uncertainty-aware stop-token boosting with structural guardrails, it achieves significant latency gains (up to 68%) without the "mid-sentence cutoff" common in naive truncation or simple token caps.

What Problem Does This Paper Address?

LLMs suffer from a "verbosity bias," often generating long-winded responses that increase inference costs and latency. While users can prompt a model to "be concise," this is notoriously unreliable—models often ignore the constraint or, worse, stop abruptly in the middle of a sentence.

Existing solutions usually fall into two camps: expensive retraining (like RLHF for conciseness) or crude inference-time hacks (like hard token limits). The former is resource-intensive, while the latter frequently produces broken, unreadable outputs. The authors of this paper sought a middle ground: a lightweight, training-free method to enforce conciseness that respects the structural integrity of language.

Methods

MinimalLM operates as a three-stage pipeline that intervenes during the decoding process without modifying the underlying model weights.

  1. Response Guidance: A lightweight system instruction encourages the model to plan a concise response (targeting ~200 words) and prioritize important information.
  2. Controlled Stopping (The Core Innovation): During decoding, MinimalLM uses a custom logits processor to adjust the probability of "stop tokens." This is governed by three specific gates:
  3. Length-aware Boost: As the response approaches a target length, the system gradually increases the logit scores for stop tokens.
  4. Entropy Gate: To avoid forcing a stop when the model is in the middle of a highly predictable sequence (and thus likely important content), the boost only activates when the next-token entropy is within a specific "uncertainty band."
    • Completeness Gate: This acts as a structural guardrail. It only allows the stopping intervention to trigger if the model is at a valid boundary, such as the end of a sentence, a closed code block, or balanced brackets.
  5. Final Check: If a response hits a hard maximum token cap, a post-processing step trims the text back to the last recognized complete sentence.
The three stages of MinimalLM: Guidance, Controlled Stopping, and Final Check.

Datasets

The researchers evaluated MinimalLM using three open-source models (Mistral-7B, Llama-3.1-8B, and Qwen2.5-3B) across a total of 330 questions drawn from:

  • MT-Bench: 80 first-turn prompts.
  • AlpacaEval: 150 prompts.
  • Arena-Hard-Auto: 100 prompts.
  • Domain-Specific Set: A custom collection of questions spanning eight categories (Healthcare, Law, Programming, etc.) to test generalization.

In total, the team generated and analyzed 5,940 responses to compare MinimalLM against baselines and various ablation configurations.

Key Insights and Ideas

  • Stopping vs. Skipping: The authors note that because LLMs are autoregressive, you cannot simply "skip" redundant parts of a plan. You must decide when to stop the generation entirely.
  • Structural vs. Semantic Completeness: A major "aha" moment in the paper is the distinction between these two. MinimalLM guarantees structural completeness (no broken sentences), but the authors acknowledge a trade-off with semantic completeness (whether every single requested detail was covered).
  • The Failure of Naive Stopping: Without structural guardrails, early stopping mechanisms terminated responses mid-sentence in nearly 60% of cases. MinimalLM reduced this to 0% in their primary evaluation.
  • Prompting and Decoding are Complementary: Prompting helps the model plan to be short, while decoding-time intervention enforces the budget when the plan fails.
Figure 2: Example responses from Mistral-7B to a ques- tion requesting five examples, displayed in our demon- stration interface (intermediate text omitted). Left: with- out guardrails, the stopping mechanism terminates mid- sentence during the third example (390 tokens). Right: MinimalLM provides all five examples and ends at a complete sentence (421 tokens), compared with 618 tokens for the baseline.

Key Results

  • Token Reduction: MinimalLM reduced generated tokens by an average of 47.6%.
  • Latency Gains: Individual response times improved by 19% to 68%, depending on the model.
  • Quality Retention: Despite the massive reduction in length, MinimalLM maintained a quality score of 8.04/10 (judged by Phi-4), compared to 8.42/10 for the full-length baseline. For comparison, simple truncation at the same length scored a much lower 6.46/10.
  • Reliability: MinimalLM produced zero structurally truncated outputs, whereas the "stop-token boost" alone (without guardrails) failed in 59.3% of instances.
Figure 3: Comparison against direct truncation across 330 questions and three models. Dark curves represent baseline responses truncated to 90–30% of their original length; light curves represent the same responses trimmed to the last complete sentence. Points indicate the configurations in Table 2. At comparable token reductions, MinimalLM achieves higher judged quality and completeness than direct truncation, although it retains less content from the original baseline response.

Limitations and Future Work

  • Information Coverage: The automatic judge identified missing information in about 23% of MinimalLM responses. While often these were minor details, it highlights a fundamental trade-off between brevity and exhaustive detail.
  • Fixed Budgets: The current implementation uses a uniform target (e.g., 200 words). The authors suggest that future work should involve adaptive budgets that change based on the complexity of the user's request.
  • Model Scale: The study focused on 3B-8B parameter models. How these techniques scale to frontier models (70B+) or how they interact with different decoding strategies (like beam search) remains an open question.
  • Serving Throughput: While individual latency drops, the authors noted that in static batching scenarios, the savings are less pronounced because the batch only finishes when the longest response is done. Continuous batching is recommended to realize the full efficiency gains.

Summary updated

Abstract

Large language models (LLMs) often generate responses longer than necessary, increasing inference latency and computational cost. While prompting can encourage conciseness, controlling output length without compromising response completeness remains challenging. Aggressive early stopping can reduce generation costs but risks terminating responses prematurely. We introduce MinimalLM, a training-free framework for reliable verbosity control that combines uncertainty-aware, length-dependent stop-token adjustment with structural completeness guardrails. MinimalLM separates the decision of when to stop from whether stopping is structurally appropriate, coordinating response-level instructions, decoding-time intervention, and completion checks without modifying model parameters or requiring auxiliary models. We evaluate MinimalLM across three open-source LLMs and 330 questions, producing 5,940 responses across baseline and ablation configurations. MinimalLM reduces generated tokens by 47.6%, accelerates individual responses by 19-68%, and eliminates structurally truncated outputs in the primary evaluation. Despite substantial token savings, it achieves an average quality score of 8.04/10, compared with 8.42/10 for unmodified models and 6.46/10 for length-matched truncation. Evaluations across diverse subject domains (such as healthcare, programming, law, and science) demonstrate robust efficiency gains, while ablations establish the complementary roles of decoding control and completeness constraints. Our findings highlight that efficient generation requires more than producing fewer tokens: it requires controlling when generation ends while preserving response integrity. MinimalLM offers a lightweight, training-free approach toward this goal.
BibTeX
@misc{alyosify2026say,
  author = {Mahmoud Alyosify and Mirna Imbabi and Eman Elkhamisi and Rahatara Ferdousi},
  title = {{Say Less, Mean Better: Training-Free Verbosity Control for Language Models}},
  year = {2026},
  eprint = {2611.04107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04107},
}
Figure