Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)

Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv, Judah Goldfeder, Sanjay BasuarXiv 2026 · arXiv

TL;DR

Long-context LLM generation often fails at high temperatures not because of poor training, but because standard samplers (like top-$p$) let in thousands of low-probability "junk" tokens. By switching to samplers that maintain a small, stable candidate pool—specifically top-$n\sigma$—models can maintain high performance and agentic reasoning even at extreme temperatures without any additional training.

What Problem Does This Paper Address?

It is a well-known frustration in the AI community: language models tend to "fall apart" the longer they generate. They fall into repetitive loops, lose track of complex instructions, or simply start outputting "word salad." Traditionally, researchers have tried to fix this through expensive long-context fine-tuning or architectural changes.

The authors of this paper argue that we are looking at the wrong end of the pipeline. They posit that long-context failure is often a sampling problem. When we increase "temperature" ($T$) to make models more creative or diverse, standard truncation rules like top-$p$ (Nucleus sampling) and top-$k$ fail to filter out the "tail" of the probability distribution. As the distribution flattens at high temperatures, these samplers admit a massive influx of low-probability tokens that push the model out of its training distribution, leading to a total collapse in quality.

Methods

The researchers conducted a massive sweep, testing 29 open-weight models (ranging from 1B to 122B parameters) across 133 combinations of sampler settings and temperatures.

They compared standard samplers against "full-distribution-aware" samplers:

  • Standard: Top-$k$ (fixed number of tokens) and Top-$p$ (tokens covering a cumulative probability mass).
  • Distribution-Aware: Min-$p$, P-less, Top-H, and the paper's primary focus, Top-$n\sigma$.

Top-$n\sigma$ is particularly interesting because it is mathematically invariant to temperature. It keeps tokens whose "logits" (the raw scores before softmax) fall within $n$ standard deviations of the maximum logit. Because temperature divides all logits by the same constant, the set of tokens selected by top-$n\sigma$ remains identical regardless of the heat; only their relative probabilities change.

The authors also performed a "causal test" by reordering operations: truncating the candidate pool at $T=1$ and then applying higher temperatures to the remaining tokens.

Datasets

The study utilized a comprehensive suite of benchmarks to test different aspects of long-context generation:

  • Agentic Coding: SWE-bench Verified (resolving real GitHub issues).
  • Long Reasoning: LiveCodeBench v6, AIME, HMMT 2025–26, and MATH-500.
  • Long-form Writing: LongBench-Write, WritingBench, and Arena-Write.
  • Long-context Reading: LongBench v2 and multi-key "Needle in a Haystack" retrieval up to 1 million tokens.
  • Stress Testing: A "Forced 16K" protocol where models were forced to generate long fiction/non-fiction sequences to observe where they break.

Key Insights and Ideas

  • The Candidate Pool Explosion: At $T=1$, top-$p$ might only keep 4 tokens. At $T=2$, that same setting can let in over 50,000 tokens. This "noise" is what causes model collapse.
  • Pool Size is the Best Predictor: The researchers found a Spearman correlation of -0.77 between the size of the candidate pool and the judged quality of the output. Smaller, tighter pools lead to better long-term coherence.
  • Sampler Invariance: Figure 1(a) illustrates how top-$n\sigma$ keeps the same three words as temperature rises, while top-$p$ explodes to 51,845 tokens.
Figure 1: (a) Llama-3.1-8B-Instruct partway through a science-fiction story (the step is picked by a fixed rule, Appendix H). The box shows the text so far, with the three continuations top-nσ keeps highlighted. The bars are the model’s ten likeliest next tokens, each dot column marks the tokens one rule keeps, and the number under a column is that rule’s total. Going from temperature 1 to 2 flattens the bars. Top-nσ still keeps the same three words and min-p only grows from 4 to 8, but top-p 0.9 now needs 51,845 tokens, most of the 128K vocabulary, to reach 90% of the probability. (b) At temperature 2, Qwen3.5 agents using top-p, top-k or Top-H resolve no SWE-bench Verified issues, while top-nσ agents resolve 46–62% (100 issues per group, dashes mark samplers that were not run on the fresh issue sets).
  • Order Matters: Most inference engines (vLLM, HuggingFace) apply temperature before truncation. The authors show that applying temperature after truncation (as Llama.cpp does) prevents many samplers from collapsing.

Key Results

  • Coding Agents: On Qwen3.5 models at $T=2$, agents using top-$p$ or top-$k$ resolved zero GitHub issues. Using top-$n\sigma$, the same agents resolved 46% to 62% of issues.
  • Performance Stability: Across all benchmarks, top-$n\sigma$ maintained a median of 100% of its $T=1$ performance at high temperatures, whereas top-$p$ dropped to just 26%.
  • Cross-Benchmark Collapse: Figure 2 shows a clear trend: as temperature increases, standard samplers (red/orange) drift toward 0% performance, while top-$n\sigma$ (blue) stays flat.
Figure 2: Each point is a sampler’s score at the high temperature listed on the right, divided by that same sampler’s score at temperature 1 on the same models and items, so 100% (dashed line) means turning up the heat performed strongly. Dividing by each sampler’s own baseline isolates how it reacts to temperature. Top-nσ (blue circles) sits on the dashed line in every row, while top-p, top-k and Top-H drift left and hit 0% on coding agents. Arrows mark values above 150% (P-less on Qwen3.5-9B resolves 50% at temperature 2 and 33% at 1). Reasoning rows use top-p 0.95. Complete results are in Sections 5.1–5.3 and the appendices.
  • Needle Retrieval: Even in simple retrieval tasks, high temperature causes top-$p$ to fail as the context grows longer, because the model becomes less "sure" and the sampler lets in wrong digits from the tail.

Limitations and Future Work

  • Closed Models: The study is limited to open-weight models because proprietary API providers (like OpenAI or Anthropic) do not currently allow users to choose their truncation sampler.
  • LLM-as-a-Judge: While the authors used objective benchmarks (like coding tests), some writing evaluations relied on GPT-4o-level judges, which may have inherent biases.
  • Soft-Capping: The authors noted that models with "soft-capped" logits (like Gemma-2) seem more resilient to tail-noise, suggesting a potential architectural synergy with better sampling.
  • Future Directions: The authors suggest that benchmarks should always report the full sampler configuration, as the choice of sampler can be just as impactful as the choice of model weights.

Summary updated

Abstract

Open-weight language models are widely observed to get worse the longer they generate. Often, attempts to fix this involve more training. At high temperature, The sampler, i.e. the rule that picks the next token out of the model's probabilities, decides whether long and agentic generation survives and/or thrives. We test 29 open-weight models from 1B to 122B parameters on ten established benchmarks plus needle retrieval out to one million tokens, covering SWE-bench Verified coding agents, long chain-of-thought reasoning, long-form writing, long structured output and long-context reading. Samplers that keep their pool of candidate tokens small and steady as temperature rises don't lose accuracy. Samplers that let in low-probability tokens fall apart. At temperature 2, coding agents running top-p, top-k or Top-H resolve 0 of 100 GitHub issues, while the same agents running top-nsigma resolve 46 to 62 and win every issue the two samplers disagree on, on three Qwen3.5 models. Over all benchmarks top-nsigma keeps a median 100% of its temperature-1 score where top-p keeps 26%. Counting how many tokens each sampler lets through explains why. In one Llama-3.1-8B story, going from temperature 1 to 2 grows top-p's pool from 4 tokens to 51,845 while top-nsigma stays at 3. Across 133 combinations of sampler setting and temperature, the typical pool size predicts judged quality (Spearman correlation of -0.77), and when we pick the pool at temperature 1 and only apply temperature afterwards, Top-H, min-p and p-less stop collapsing. At the default temperature of 1 most samplers perform about the same, We release all code and data to the public.
BibTeX
@misc{roush2026long,
  author = {Allen G. Roush and Minh Nhat Nguyen and Ravid Shwartz-Ziv and Judah Goldfeder and Sanjay Basu},
  title = {{Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)}},
  year = {2026},
  eprint = {2611.22148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22148},
}
Figure