TL;DR
MinimalLM is a training-free framework that slashes Large Language Model (LLM) verbosity by nearly 50% while ensuring responses remain structurally complete. By combining uncertainty-aware stop-token boosting with structural guardrails, it achieves significant latency gains (up to 68%) without the "mid-sentence cutoff" common in naive truncation or simple token caps.
What Problem Does This Paper Address?
LLMs suffer from a "verbosity bias," often generating long-winded responses that increase inference costs and latency. While users can prompt a model to "be concise," this is notoriously unreliable—models often ignore the constraint or, worse, stop abruptly in the middle of a sentence.
Existing solutions usually fall into two camps: expensive retraining (like RLHF for conciseness) or crude inference-time hacks (like hard token limits). The former is resource-intensive, while the latter frequently produces broken, unreadable outputs. The authors of this paper sought a middle ground: a lightweight, training-free method to enforce conciseness that respects the structural integrity of language.
Methods
MinimalLM operates as a three-stage pipeline that intervenes during the decoding process without modifying the underlying model weights.
- Response Guidance: A lightweight system instruction encourages the model to plan a concise response (targeting ~200 words) and prioritize important information.
- Controlled Stopping (The Core Innovation): During decoding, MinimalLM uses a custom logits processor to adjust the probability of "stop tokens." This is governed by three specific gates:
- Length-aware Boost: As the response approaches a target length, the system gradually increases the logit scores for stop tokens.
- Entropy Gate: To avoid forcing a stop when the model is in the middle of a highly predictable sequence (and thus likely important content), the boost only activates when the next-token entropy is within a specific "uncertainty band."
- Completeness Gate: This acts as a structural guardrail. It only allows the stopping intervention to trigger if the model is at a valid boundary, such as the end of a sentence, a closed code block, or balanced brackets.
- Final Check: If a response hits a hard maximum token cap, a post-processing step trims the text back to the last recognized complete sentence.
Datasets
The researchers evaluated MinimalLM using three open-source models (Mistral-7B, Llama-3.1-8B, and Qwen2.5-3B) across a total of 330 questions drawn from:
- MT-Bench: 80 first-turn prompts.
- AlpacaEval: 150 prompts.
- Arena-Hard-Auto: 100 prompts.
- Domain-Specific Set: A custom collection of questions spanning eight categories (Healthcare, Law, Programming, etc.) to test generalization.
In total, the team generated and analyzed 5,940 responses to compare MinimalLM against baselines and various ablation configurations.
Key Insights and Ideas
- Stopping vs. Skipping: The authors note that because LLMs are autoregressive, you cannot simply "skip" redundant parts of a plan. You must decide when to stop the generation entirely.
- Structural vs. Semantic Completeness: A major "aha" moment in the paper is the distinction between these two. MinimalLM guarantees structural completeness (no broken sentences), but the authors acknowledge a trade-off with semantic completeness (whether every single requested detail was covered).
- The Failure of Naive Stopping: Without structural guardrails, early stopping mechanisms terminated responses mid-sentence in nearly 60% of cases. MinimalLM reduced this to 0% in their primary evaluation.
- Prompting and Decoding are Complementary: Prompting helps the model plan to be short, while decoding-time intervention enforces the budget when the plan fails.
Key Results
- Token Reduction: MinimalLM reduced generated tokens by an average of 47.6%.
- Latency Gains: Individual response times improved by 19% to 68%, depending on the model.
- Quality Retention: Despite the massive reduction in length, MinimalLM maintained a quality score of 8.04/10 (judged by Phi-4), compared to 8.42/10 for the full-length baseline. For comparison, simple truncation at the same length scored a much lower 6.46/10.
- Reliability: MinimalLM produced zero structurally truncated outputs, whereas the "stop-token boost" alone (without guardrails) failed in 59.3% of instances.
Limitations and Future Work
- Information Coverage: The automatic judge identified missing information in about 23% of MinimalLM responses. While often these were minor details, it highlights a fundamental trade-off between brevity and exhaustive detail.
- Fixed Budgets: The current implementation uses a uniform target (e.g., 200 words). The authors suggest that future work should involve adaptive budgets that change based on the complexity of the user's request.
- Model Scale: The study focused on 3B-8B parameter models. How these techniques scale to frontier models (70B+) or how they interact with different decoding strategies (like beam search) remains an open question.
- Serving Throughput: While individual latency drops, the authors noted that in static batching scenarios, the savings are less pronounced because the batch only finishes when the longest response is done. Continuous batching is recommended to realize the full efficiency gains.
Summary updated