Abstract
As Large Language Models (LLMs) are increasingly deployed to analyze massive codebases, their ability to reason over long contexts becomes critical. This paper identifies a fundamental "dissociation" in LLM performance: while models exhibit near-perfect lexical recall (retrieving code verbatim), their semantic recall (understanding and executing that code) degrades significantly when relevant logic is placed in the middle of a long input. By introducing SemTrace, a high-sensitivity benchmark designed to prevent pattern-matching shortcuts, the authors reveal that current evaluations significantly overestimate true code reasoning capabilities.
Introduction
Modern LLMs boast context windows of up to millions of tokens, promising the ability to reason across entire software repositories. However, it remains unclear if these models are truly "tracing" code logic or merely recognizing familiar algorithmic patterns from their training data. Existing benchmarks like "Needle-in-a-Haystack" (NIAH) primarily test lexical retrieval, while code reasoning benchmarks like CRUXEval may be susceptible to "cheating" via pattern matching. This study aims to isolate true semantic recall and measure its sensitivity to the position of code within long contexts.
Methodology: Isolating Semantic Recall
The researchers distinguish between the ability to reproduce code (lexical) and the ability to use that code for downstream reasoning (semantic). To test the robustness of these capabilities, they employed several techniques:
- Positional Probing: Target code snippets were embedded at 11 different locations within "distractor" contexts (irrelevant Python functions) ranging from 4k to 16k tokens.
- Counterfactual Measurement: To detect pattern-matching shortcuts, the authors systematically removed lines of code. If a model maintains high accuracy despite missing logic, it is likely relying on memorized patterns rather than the provided text.
- SemTrace Benchmark: A new, high-sensitivity dataset consisting of randomized, unpredictable arithmetic assignments. Unlike standard algorithms (e.g., quicksort), SemTrace requires the model to semantically recall every line to reach the correct output.
Results: The Great Dissociation
The study reveals a striking performance gap. Frontier models like GPT-4.1 and DeepSeek Coder V2 show near-perfect, position-independent lexical recall. However, their semantic recall is highly fragile. On the SemTrace benchmark, the median accuracy drop was a staggering 92.73% when target code moved from the edges to the middle of a 16k token context. Some models reached zero accuracy in the middle positions despite perfect performance when the code was at the very beginning or end.
The Pattern Matching Shortcut
The researchers found that existing benchmarks are often "leaky." On CRUXEval, models lost only ~44-60% accuracy when half the code was deleted, suggesting they were guessing based on common coding patterns. This is further evidenced by the "Arithmetic Shortcut": GPT-4.1 initially appeared to solve long-context SemTrace tasks, but further testing revealed it had simply memorized 2-digit addition tables. When task difficulty increased to 4-digit or 6-digit math, the model exhibited the same "lost-in-the-middle" failures as its peers.
Conclusion
This research demonstrates that current LLMs struggle with true semantic reasoning in long contexts, even when they can successfully locate the relevant code. The "lost-in-the-middle" effect is far more severe for semantic tasks than lexical ones. Future development must move beyond simple retrieval metrics and focus on robust semantic recall to ensure LLMs can reliably analyze complex, real-world codebases.
Summary updated