TL;DR
Long-context LLM generation often fails at high temperatures not because of poor training, but because standard samplers (like top-$p$) let in thousands of low-probability "junk" tokens. By switching to samplers that maintain a small, stable candidate pool—specifically top-$n\sigma$—models can maintain high performance and agentic reasoning even at extreme temperatures without any additional training.
What Problem Does This Paper Address?
It is a well-known frustration in the AI community: language models tend to "fall apart" the longer they generate. They fall into repetitive loops, lose track of complex instructions, or simply start outputting "word salad." Traditionally, researchers have tried to fix this through expensive long-context fine-tuning or architectural changes.
The authors of this paper argue that we are looking at the wrong end of the pipeline. They posit that long-context failure is often a sampling problem. When we increase "temperature" ($T$) to make models more creative or diverse, standard truncation rules like top-$p$ (Nucleus sampling) and top-$k$ fail to filter out the "tail" of the probability distribution. As the distribution flattens at high temperatures, these samplers admit a massive influx of low-probability tokens that push the model out of its training distribution, leading to a total collapse in quality.
Methods
The researchers conducted a massive sweep, testing 29 open-weight models (ranging from 1B to 122B parameters) across 133 combinations of sampler settings and temperatures.
They compared standard samplers against "full-distribution-aware" samplers:
- Standard: Top-$k$ (fixed number of tokens) and Top-$p$ (tokens covering a cumulative probability mass).
- Distribution-Aware: Min-$p$, P-less, Top-H, and the paper's primary focus, Top-$n\sigma$.
Top-$n\sigma$ is particularly interesting because it is mathematically invariant to temperature. It keeps tokens whose "logits" (the raw scores before softmax) fall within $n$ standard deviations of the maximum logit. Because temperature divides all logits by the same constant, the set of tokens selected by top-$n\sigma$ remains identical regardless of the heat; only their relative probabilities change.
The authors also performed a "causal test" by reordering operations: truncating the candidate pool at $T=1$ and then applying higher temperatures to the remaining tokens.
Datasets
The study utilized a comprehensive suite of benchmarks to test different aspects of long-context generation:
- Agentic Coding: SWE-bench Verified (resolving real GitHub issues).
- Long Reasoning: LiveCodeBench v6, AIME, HMMT 2025–26, and MATH-500.
- Long-form Writing: LongBench-Write, WritingBench, and Arena-Write.
- Long-context Reading: LongBench v2 and multi-key "Needle in a Haystack" retrieval up to 1 million tokens.
- Stress Testing: A "Forced 16K" protocol where models were forced to generate long fiction/non-fiction sequences to observe where they break.
Key Insights and Ideas
- The Candidate Pool Explosion: At $T=1$, top-$p$ might only keep 4 tokens. At $T=2$, that same setting can let in over 50,000 tokens. This "noise" is what causes model collapse.
- Pool Size is the Best Predictor: The researchers found a Spearman correlation of -0.77 between the size of the candidate pool and the judged quality of the output. Smaller, tighter pools lead to better long-term coherence.
- Sampler Invariance: Figure 1(a) illustrates how top-$n\sigma$ keeps the same three words as temperature rises, while top-$p$ explodes to 51,845 tokens.
- Order Matters: Most inference engines (vLLM, HuggingFace) apply temperature before truncation. The authors show that applying temperature after truncation (as Llama.cpp does) prevents many samplers from collapsing.
Key Results
- Coding Agents: On Qwen3.5 models at $T=2$, agents using top-$p$ or top-$k$ resolved zero GitHub issues. Using top-$n\sigma$, the same agents resolved 46% to 62% of issues.
- Performance Stability: Across all benchmarks, top-$n\sigma$ maintained a median of 100% of its $T=1$ performance at high temperatures, whereas top-$p$ dropped to just 26%.
- Cross-Benchmark Collapse: Figure 2 shows a clear trend: as temperature increases, standard samplers (red/orange) drift toward 0% performance, while top-$n\sigma$ (blue) stays flat.
- Needle Retrieval: Even in simple retrieval tasks, high temperature causes top-$p$ to fail as the context grows longer, because the model becomes less "sure" and the sampler lets in wrong digits from the tail.
Limitations and Future Work
- Closed Models: The study is limited to open-weight models because proprietary API providers (like OpenAI or Anthropic) do not currently allow users to choose their truncation sampler.
- LLM-as-a-Judge: While the authors used objective benchmarks (like coding tests), some writing evaluations relied on GPT-4o-level judges, which may have inherent biases.
- Soft-Capping: The authors noted that models with "soft-capped" logits (like Gemma-2) seem more resilient to tail-noise, suggesting a potential architectural synergy with better sampling.
- Future Directions: The authors suggest that benchmarks should always report the full sampler configuration, as the choice of sampler can be just as impactful as the choice of model weights.
Summary updated