TL;DR
When Large Language Models (LLMs) are uncertain about an answer, they often fail to be concise—even when explicitly instructed to be brief—hedging their bets by repeating the prompt, listing multiple options, or padding their response with filler. The authors term this Verbosity Compensation (VC) and show that verbose answers are significantly less accurate and harbor higher predictive uncertainty across 14 frontier models. Because verbosity reliably flags model confusion, a simple cascade router that hands off verbose outputs to stronger LLMs cuts verbosity and boosts task accuracy while minimizing API costs.
What Problem Does This Paper Address?
When humans are unsure of an answer, they tend to stall, hesitate, and use filler words. Language models exhibit a strikingly similar fallback behavior: when prompted to provide concise, direct answers (e.g., "answer in 3 words or fewer"), uncertain models often ignore the length constraint and produce flabby, evasive responses.
While prior work has pointed out length biases stemming from Reinforcement Learning from Human Feedback (RLHF), few have systematically explored why models produce unprompted verbosity on factual questions or how it relates to internal model confidence. This paper formalizes and analyzes Verbosity Compensation (VC)—the generation of compressible, low-information tokens under concise prompting—and investigates how verbosity, correctness, and uncertainty intersect.
Methods
-
Concise Question Answering (CQA) Formulation:
The authors prompt models using reinforced instructions requiring extreme conciseness (e.g., "Using a single phrase rather than a sentence. Please answer in 3 words."). A response $r$ is labeled as exhibiting VC via a detector $V(x, y, r) = \mathbb{I}(|r| > 3)$ on tasks where the ground truth is strictly under 4 words. -
Quantifying Gaps and Uncertainty:
- Performance Gap ($\Delta$ and $\delta$): The authors evaluate token recall on concise versus verbose splits. $\Delta$ measures the raw recall drop of verbose answers relative to concise ones, while $\delta$ normalizes this metric across datasets with differing baseline accuracies.
- Uncertainty Metrics: Open-source models are evaluated via response perplexity; closed-source black-box models are evaluated via graph Laplacian eigenvalue summation across multiple sampled outputs.
-
Taxonomy of Failure Modes:
Human annotation groups non-concise incorrect responses into five categories: repeating questions, enumerating multiple guesses, ambiguity (hedging vaguely), verbose details, and verbose formatting. -
Cascade Model Selection (CaSel):
Leveraging verbosity as a reference-free proxy for error, the authors propose a cascade algorithm. A smaller, cheaper model generates an initial answer; if the output is flagged as verbose ($|r| > 3$), it is discarded, and a larger model is queried instead.
┌──────────────────────┐
│ Query + Concise Prompt│
└──────────┬───────────┘
│
▼
┌─────────────────┐
│ Weaker Model │
└────────┬────────┘
│
Output Length r
│
[ |r| ≤ 3 ] ┴ [ |r| > 3 (Verbose) ]
│ │
▼ ▼
┌──────────────┐ ┌─────────────────┐
│ Return Final │ │ Stronger Model │
│ Response │ └────────┬────────┘
└──────────────┘ │
▼
┌─────────────────┐
│ Return Response │
└─────────────────┘
Datasets
The authors evaluate on 5 question-answering benchmarks, filtering for instances where ground-truth answers are under 4 words:
- Long-context Knowledge Extraction:
- Qasper: Short/medium-length NLP research papers (~4.1k words).
- LongBench: Multi-hop questions aggregated from HotpotQA, MuSiQue, and 2WikiMultihopQA (~9.5k words).
- NarrativeQA: Very long contexts over books and movie scripts (~70.3k words).
- NaturalQuestions-30 (NQ30): Evaluates "lost-in-the-middle" challenges by placing target information in the middle of 30 distractor documents (~3.6k words).
- Reasoning-based Understanding:
- MMLU (Modified): Multiple-choice questions converted into open-ended questions by treating the choices as general hints rather than discrete options.
- (Supplementary tasks include Chain-of-Thought reasoning steps and Natural-Plan trip planning).
Key Insights and Ideas
- Verbosity Flags Uncertainty: First-token softmax distributions reveal that when a model is unsure, the probability mass flattens across candidate tokens. Instead of picking a high-information token, the model defaults to safe "filler" tokens (e.g., repeating "The average sentence length is..." or starting with punctuation). These filler tokens are syntactically incomplete, forcing the auto-regressive generation loop to generate longer, rambling explanations.
- Verbosity $\neq$ Veracity: Across almost all models and benchmarks, verbose responses scored substantially worse than concise ones. For example, Llama-3-70B experienced a 27.61% recall drop on verbose answers compared to concise ones on Qasper.
- Model Scale Does Not Eradicate VC: While larger context windows and higher Chatbot Arena Elo ratings mitigate VC in long-context retrieval (Qasper, LongBench, NarrativeQA), they do not diminish the relative performance gap ($\delta$) on complex reasoning benchmarks like MMLU or NQ30.
- Length as a Free Quality Filter: Because verbosity strongly correlates with model uncertainty and hallucinations, response token length serves as a near-zero-cost uncertainty estimator, outperforming perplexity-based routing on cost-vs-quality trade-off curves.
Key Results
- Pervasive Behavior: VC occurs frequently across all 14 evaluated LLMs. Open-source models average a 39.80% VC rate, while closed-source models average 28.96%. Mistral-7B was verbose on 74.19% of instances, while even GPT-4o exhibited a 50.40% VC frequency on NarrativeQA.
- Consistent Accuracy Drops: Across all models, concise answers averaged 62.76% recall on Qasper, whereas verbose answers averaged only 42.42% (an average $\Delta$ gap of $+20.34\%$).
- Effective Cascade Mitigation:
- Mistral-7B $\rightarrow$ GPT-4o reduced overall VC frequency from 74.19% (Mistral) and 29.39% (GPT-4o) down to 16.16%, while slashing the number of useless tokens generated from 663 down to 86 on Qasper.
- In LLM routing experiments, the verbosity-based cascade achieved higher F1 scores at lower average API costs than both random selection baselines and perplexity-based uncertainty routing.
Limitations and Future Work
- Heuristic VC Detection: The primary detector relies on a hard token-count threshold ($|r| > 3$) enabled by filtering ground truth to short answers. While effective for CQA, open-ended tasks require more sophisticated semantic compression algorithms to identify non-essential tokens.
- Prompt Sensitivity: Although the authors validated several concise prompt formulations, models still vary in instruction-following compliance.
- Future Directions: The authors suggest using the empirical link between verbosity and predictive entropy to design decoding strategies or fine-tuning objectives that penalize hesitation tokens, thereby directly reducing model hallucinations and server latency.
Summary updated