Code assistants are increasingly given whole repositories as context, yet adding more repository files often makes them worse. A common explanation is "lost in the middle": language models use information at the start and end of a long prompt far better than information buried in between. This proposal tests a sharper hypothesis about what the model does when it fails to read the middle. We think it does not simply err at random. It often falls back on code it memorized during training, reproducing the behavior of a familiar implementation even when the repository in front of it says something different. If that is right, positional failure and memorization leakage are one problem seen from two sides, and current benchmarks, which score only pass or fail, cannot tell them apart.
The method is a counterfactual repository. We take functions a model is known to have memorized in function, not just in wording. We embed them in realistic multi-file repositories and then edit the in-context definitions so the correct answer differs from the memorized one, for example by changing a hard-coded commission rate or the semantics of a helper. Tests that run both versions then sort every generation into one of three classes: it followed the context, it reproduced the memorized original, or it failed some other way. Varying where the edited definition sits in a long prompt, and how the repository is serialized into tokens, gives the first direct measurement of memory override as a function of position and of structural distance in the code's dependency graph.
Core Claim
In repository-level code generation, part of the middle-of-context accuracy drop is memorization-attributable override. When a needed definition sits in the middle of a flattened prompt, a model that saw similar code in training emits execution-verified behavior matching the training version rather than the context version, and does so more often than an otherwise identical model that never saw that code. We further claim that override depends on how far the definition is from its use in the dependency graph as well as on raw token position. If so, the information lost when code is flattened into a token sequence and the lost-in-the-middle effect interact rather than add.
Each part is falsifiable. The override rate may turn out flat across position, in which case middle failures are generic errors. It may not differ between exposed and unexposed models, in which case they are not memorization. Or token position may fully explain it once graph distance is controlled.
Why This Is Not Already Done
The closest work is Detecting Functional Memorization in Code Language Models. It compares an Olmo-3-32B checkpoint midtrained on the CraneCode dataset against a reference checkpoint that never saw it, and finds generations with low textual overlap but identical behavior. Its headline rates are 0.58% verbatim, 3.9% judged functional by an LLM, and 0.28% execution-verified clones. One example is proprietary commission rates for Sofia, Varna and Plovdiv, rewritten from dictionary iteration into tuple indexing. That paper motivates our design and supplies the exposed-versus-reference contrast we reuse. But it prompts with isolated function signatures. It never puts the model in a position where context and memory disagree, and it never varies context length or placement. Its sandbox also executed only about 15% of samples, because real functions depend on class state, external services and file systems.
On the positional side, Lost in the Middle established the U-shaped curve on document QA and key-value lookup. Serial Position Effects of Large Language Models generalized it. SWE-bench documented the repository version of the problem, where more retrieved files means fewer resolved issues. None of these asks what the model writes instead.
Two lines of work come closest to the mechanism, both on natural-language knowledge conflicts. Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement shows that contextual information peaks in middle layers and is overridden by stored knowledge in deeper ones. Parameters vs. Context detects context-memory conflicts from entropy differences. Neither touches code, execution or position.
Structure-aware systems are also silent on this question. Code Graph Model (CGM) and DyCoder improve task accuracy by passing the model a repository's dependency graph, or by building partial call graphs on demand. They report only resolution rate and Pass@1.
Our contribution is therefore not a new memorization detector or a new long-context method. It is an evaluation design in which memorization becomes a ground-truth signal of whether the model read a specific region of its context. Because we write the edits and the distinguishing tests, the design also sidesteps the execution-coverage problem.
Research Program
Building counterfactual repositories. We start from the filtered CraneCode subset of 7,422 business-logic functions, and rank functions by the exposed-minus-reference gap in likelihood and in the LLM-judge rating. The execution-verified pool from the prior paper is small, roughly twenty functions at 0.28%, so ranking is needed to get enough candidates.
For each selected function we apply a semantics-changing edit to its in-context definition, or to a helper it calls. Typical edits change a constant, invert a branch condition, or reorder arguments. We then synthesize self-contained distinguishing tests whose outputs differ between the original and the edited version. Editing and test generation are automated with an LLM and checked by running both versions. Each function is placed in a synthetic repository with realistic filler files, so the full context can be grown from 8k to 64k tokens.
As a lower-memorization control, we apply the same edit procedure to repositories from CrossCodeEval, which contains repositories created after March 2023, and from DevEval. Whether these repositories appear in Olmo's training data must be checked with its open data index and with MIN-K% PROB from Detecting Pretraining Data from Large Language Models. We do not assume the answer.
A controlled exposure arm. Natural memorization is rare and its training history is uncontrolled, so we add a small-scale arm. Following the twin-run design of Memorization Dynamics of Fill-in-the-Middle Pretraining, we continue pretraining a roughly 3B open code model on a corpus seeded with synthetic business-rule modules repeated 1 to 128 times, alongside a twin run without them. This gives a dose-response curve, override rate against number of training repetitions, with exact knowledge of which code was seen. It also lets us compare models trained only left-to-right with models trained on fill-in-the-middle, an objective in which the model completes a missing span given both the code before and after it. That paper found such models strongly anchored to the preceding code. We test whether that anchoring makes edits placed after the cursor especially prone to override.
Disentangling token position from structural distance. We cross two factors. The first is where the edited definition appears in the flattened prompt: start, several middle depths, or end. The second is its distance from the generation site in the call graph: a direct callee, or reached through two or three intermediate helpers that are also in context. Filler files are shuffled so the two factors vary independently.
We then compare three ways of serializing the same files. The first is the naive retrieval order. The second is a dependency-ordered serialization, a topological ordering that places each definition next to its callers, without any oracle knowledge of which definition was edited. The third is the graph-aware attention input used by CGM, run where its released models allow.
Outcome measure. For each condition we report three numbers. The faithful rate is the fraction of generations passing the edited-version tests. The override rate is the fraction passing the original-version tests. The memorization-attributable override is the override rate of the exposed model minus that of the reference model on identical prompts. That difference is the quantity the core claim is about.
Baselines
The naive reference point is the exposed and reference models given the counterfactual repository in naive retrieval order with greedy decoding. This fixes the size of the problem, and the reference model's override rate is the floor that chance coincidence alone produces.
The strongest positional competitor is Positional Contrastive Decoding from Mitigating Posterior Salience Attenuation. It contrasts normal logits against those of a deliberately "near-sighted" pass to restore the salience of distant tokens, and reports InfiniteBench 8k key-value retrieval accuracy of 79.0% versus 72.0% greedy, and RULER 16k variable-tracking F1 of 69.04 versus 64.40, on Llama-3-8B-262k. It is the strongest competitor because its account, that the right answer is encoded but outcompeted by nearby tokens, is the main rival to our override account.
The strongest memory-versus-context competitor is CK-PLUG from Parameters vs. Context. It directly targets conflicts between context and stored knowledge, which is exactly the regime our edits create. We also include layer enhancement (CaLE), which Exploiting Contextual Knowledge reports raising CounterFact exact match from 54.32% to 74.98% on Llama2-7B.
There is no reported result on Lacuna for any of these methods on code, nor for any context-versus-memory override rate in code. The numbers we produce will be the first.
All decoding methods run on the same models, prompts and test suites. Each gets the same hyperparameter sweep, five values of its mixing coefficient, chosen on a held-out split of repositories. We report latency alongside accuracy, because both contrastive methods require extra forward passes.
The key ablation compares dependency-ordered serialization against a random permutation that keeps the token length and the position of the edited definition fixed. Only the adjacency of related code changes. It isolates whether structure matters beyond position. We also run an oracle recency condition that moves the edited definition to the end of the prompt. Beating a baseline means a statistically significant reduction in memorization-attributable override at middle positions, using paired bootstrap over functions, without a drop in faithful rate at edge positions.
Why not just…?
Why not just put the relevant code at the end of the prompt, or repeat the query? If oracle recency eliminates override, this is a real threat to the structural half of the claim, and we will report it as such. Two cautions apply, though. In deployment, the retriever does not know which file contains the decisive definition; DyCoder shows that similarity-based retrieval misses utilities whose text does not resemble the call site. Lost in the Middle also found that repeating the query at both ends helps synthetic lookup but not harder reasoning. The oracle condition is our upper bound, and the gap between it and dependency ordering, which needs no oracle, is the honest measure of what structure buys.
Why not just measure Pass@1 at different positions on DevEval or SWE-bench? That would show where models fail, which DevEval already shows coarsely with 18.03% versus 44.83% Pass@1 without and with oracle context. It cannot show whether a failure is a fallback to memory or an ordinary mistake. Only tests that run both the original and the edited version separate the two, so this baseline is subsumed by our measurement rather than competing with it.
Why not just apply PCD or CK-PLUG and call the problem solved? They may well work, and running them is part of the study. The informative outcome is which one works. If PCD reduces generic middle errors but leaves override untouched, while CK-PLUG does the reverse, salience loss and memory override are separable mechanisms. If one method removes both, the problem has a single cause and that method is the fix.
Why not just detect memorization with an LLM judge or MIN-K% PROB? The functional-memorization paper reports that LLM judges are prompt-sensitive and expensive. MIN-K% PROB is known to degrade on paraphrased text. Neither measures whether the model used its context. Constructed tests that run both versions are exact, and they work on all of our tasks, unlike HyClone's roughly 15% coverage on real code.
Decisive Checks
Three results would settle the core claim. First, on Olmo-3-32B, the memorization-attributable override rate must be positive and must peak at middle positions, with a U-shape mirroring the faithful rate. A flat curve rejects the claim. Second, in the controlled-exposure arm, override must rise with the number of training repetitions at middle positions but much less at edge positions. Without that interaction, position and memory act independently. Third, with token position held fixed, override must grow with call-graph distance, and dependency-ordered serialization must recover a meaningful part of the gap to oracle recency. If it does not, structural information loss is not part of the story, and the paper reduces to a cleaner positional measurement.
Each check can come out negative and still be publishable, because the counterfactual-repository suite is reusable as a context-faithfulness benchmark for code models.
References
- Detecting Functional Memorization in Code Language Models
- Lost in the Middle: How Language Models Use Long Contexts
- Serial Position Effects of Large Language Models
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement
- Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Models
- Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks
- Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generation
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
- DevEval: Evaluating Code Generation in Practical Software Projects
- Detecting Pretraining Data from Large Language Models
- Memorization Dynamics of Fill-in-the-Middle Pretraining
- Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding