Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning

Adam Storek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava, Suman JanaarXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) are increasingly deployed to analyze massive codebases, their ability to reason over long contexts becomes critical. This paper identifies a fundamental "dissociation" in LLM performance: while models exhibit near-perfect lexical recall (retrieving code verbatim), their semantic recall (understanding and executing that code) degrades significantly when relevant logic is placed in the middle of a long input. By introducing SemTrace, a high-sensitivity benchmark designed to prevent pattern-matching shortcuts, the authors reveal that current evaluations significantly overestimate true code reasoning capabilities.

Introduction

Modern LLMs boast context windows of up to millions of tokens, promising the ability to reason across entire software repositories. However, it remains unclear if these models are truly "tracing" code logic or merely recognizing familiar algorithmic patterns from their training data. Existing benchmarks like "Needle-in-a-Haystack" (NIAH) primarily test lexical retrieval, while code reasoning benchmarks like CRUXEval may be susceptible to "cheating" via pattern matching. This study aims to isolate true semantic recall and measure its sensitivity to the position of code within long contexts.

Methodology: Isolating Semantic Recall

The researchers distinguish between the ability to reproduce code (lexical) and the ability to use that code for downstream reasoning (semantic). To test the robustness of these capabilities, they employed several techniques:

  • Positional Probing: Target code snippets were embedded at 11 different locations within "distractor" contexts (irrelevant Python functions) ranging from 4k to 16k tokens.
  • Counterfactual Measurement: To detect pattern-matching shortcuts, the authors systematically removed lines of code. If a model maintains high accuracy despite missing logic, it is likely relying on memorized patterns rather than the provided text.
  • SemTrace Benchmark: A new, high-sensitivity dataset consisting of randomized, unpredictable arithmetic assignments. Unlike standard algorithms (e.g., quicksort), SemTrace requires the model to semantically recall every line to reach the correct output.
Model accuracy as a function of removed code lines, highlighting reliance on pattern matching.

Results: The Great Dissociation

The study reveals a striking performance gap. Frontier models like GPT-4.1 and DeepSeek Coder V2 show near-perfect, position-independent lexical recall. However, their semantic recall is highly fragile. On the SemTrace benchmark, the median accuracy drop was a staggering 92.73% when target code moved from the edges to the middle of a 16k token context. Some models reached zero accuracy in the middle positions despite perfect performance when the code was at the very beginning or end.

Lexical vs. Semantic Recall across different positions and distractors.

The Pattern Matching Shortcut

The researchers found that existing benchmarks are often "leaky." On CRUXEval, models lost only ~44-60% accuracy when half the code was deleted, suggesting they were guessing based on common coding patterns. This is further evidenced by the "Arithmetic Shortcut": GPT-4.1 initially appeared to solve long-context SemTrace tasks, but further testing revealed it had simply memorized 2-digit addition tables. When task difficulty increased to 4-digit or 6-digit math, the model exhibited the same "lost-in-the-middle" failures as its peers.

Conclusion

This research demonstrates that current LLMs struggle with true semantic reasoning in long contexts, even when they can successfully locate the relevant code. The "lost-in-the-middle" effect is far more severe for semantic tasks than lexical ones. Future development must move beyond simple retrieval metrics and focus on robust semantic recall to ensure LLMs can reliably analyze complex, real-world codebases.

Summary updated

Abstract

Large language models (LLMs) are increasingly deployed for understanding large codebases, but whether they understand operational semantics of long code context or rely on pattern matching shortcuts remains unclear. We distinguish between lexical recall (retrieving code verbatim) and semantic recall (understanding operational semantics). Evaluating 10 state-of-the-art LLMs, we find that while frontier models achieve near-perfect, position-independent lexical recall, semantic recall degrades severely when code is centrally positioned in long contexts. We introduce semantic recall sensitivity to measure whether tasks require understanding of code's operational semantics vs. permit pattern matching shortcuts. Through a novel counterfactual measurement method, we show that models rely heavily on pattern matching shortcuts to solve existing code understanding benchmarks. We propose a new task SemTrace, which achieves high semantic recall sensitivity through unpredictable operations; LLMs' accuracy exhibits severe positional effects, with median accuracy drops of 92.73% versus CRUXEval's 53.36% as the relevant code snippet approaches the middle of the input code context. Our findings suggest current evaluations substantially underestimate semantic recall failures in long context code understanding.
BibTeX
@misc{storek2025sense,
  author = {Adam Storek and Mukur Gupta and Samira Hajizadeh and Prashast Srivastava and Suman Jana},
  title = {{Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning}},
  year = {2025},
  eprint = {2505.13353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.13353},
}
Figure