Abstract Recent advancements in Large Language Models (LLMs) have led to increasingly large context windows, often exceeding one million tokens. However, standard benchmarks like Needle-in-a-Haystack (NIAH) primarily measure retrieval from irrelevant noise rather than the integration of dense, relevant information. We introduce NeedleChain, a benchmark designed to evaluate "intact context comprehension" by requiring models to synthesize a chain of interconnected facts where every token is query-relevant. Our findings reveal a significant fragility in state-of-the-art models: even advanced systems like GPT-4o struggle with reasoning chains as short as 200 tokens, particularly when information is presented in non-linear or backward sequences.
Introduction
The prevailing narrative suggests that LLMs have "solved" long-context processing. This perception is largely driven by the success of models on the NIAH paradigm, where a single "needle" of information is buried within a "haystack" of irrelevant text. We argue that this overestimates true understanding. In critical domains such as legal analysis or medical diagnosis, the context is rarely noise; it is a series of interdependent facts. NeedleChain shifts the focus from simple retrieval to complex integration, testing whether a model can maintain a perfect mental model of a context where every sentence matters.
Methodology
NeedleChain consists of factual statements (needles) that form a causal reasoning chain, typically involving salary comparisons (e.g., "Alice earns $100," "Bob earns twice as much as Alice"). To test different cognitive loads and logical biases, we developed three variants of the chain:
- Forward Chain: Facts are presented in logical, chronological order (A → B → C).
- Backward Chain: Facts are presented in reverse logical order (C → B → A).
- Mixed Chain: Facts are shuffled randomly throughout the context.
For comparison, we also introduced NeedleStack, a parallel NIAH-style benchmark. In NeedleStack, the same factual components are used, but all but one needle are treated as irrelevant noise, isolating the model's ability to retrieve a single fact versus reasoning across a chain.
The Retrieval vs. Reasoning Gap
Our evaluation of models including Qwen2.5-32B, Llama3.3-70B, and GPT-4o reveals a staggering performance gap. While these models achieve near 100% accuracy on NeedleStack (retrieval) even at high token counts, their performance on NeedleChain (reasoning) collapses rapidly as the number of needles ($k$) increases.
As shown in Figure 1, GPT-4o's accuracy on the Backward Chain drops to 78.5% at just $k=20$ (~200 tokens). By $k=200$ (~2,000 tokens), accuracy for most models plummets toward zero. This "1K Token Wall" suggests that "needle omission"—simply forgetting a link in the chain—becomes the dominant failure mode once context density increases.
Key Insights
- Directional Vulnerability: Models perform significantly better on Forward Chains than Backward Chains. This suggests a "logical bias" stemming from the left-to-right nature of autoregressive training data, making it difficult for models to integrate information presented in reverse order.
- Logically "Lost-in-the-Middle": While previous studies noted models lose information in the physical middle of long texts, NeedleChain shows they also get logically lost. In Mixed Chains, models consistently fail to integrate the intermediate steps of a logic chain, regardless of their physical position in the prompt.
- RoPE Contraction: We propose a training-free strategy called RoPE contraction. Unlike RoPE extension, which stretches context windows, contraction "sharpens" positional embeddings at inference time. This sharpening significantly improves a model's ability to distinguish and track multiple dense facts.
Conclusion
NeedleChain highlights that current LLMs are far from mastering long-context reasoning. The ability to retrieve a single fact from a haystack does not equate to the ability to synthesize a chain of needles. Future research should prioritize "intact comprehension"—ensuring models can reliably use 10,000 tokens of dense information before pursuing million-token windows that they cannot yet fully integrate.
Summary updated