A repository is too large to fit into a language model's prompt. Two remedies are available. The model can read a pruned or retrieved slice of the repository in its context window, or it can memorize the repository by briefly training on it at inference time. That second option is called test-time training: gradient updates are run on the test input before the model answers. Memorization escapes the context limit, but each gradient step adds delay before the first output token. It also stores code in the weights without any record of where each fact came from. This note proposes to measure what that trade buys and what it costs. We split repository code generation into tiers of increasing difficulty: recalling a distant signature, tracing how dependencies connect, and writing a correct function. We add "unsolvable" controls in which a required definition has been deleted from the repository. We then compare reading-based and memorizing-based remedies at matched wall-clock latency.
The central bet is that memorization repairs the lowest tier, recall, more cheaply than reading does. We also bet that it reproduces a known pathology of narrow fine-tuning: the model starts to assume that any plausible-looking function exists. If both hold, the right choice between reading and memorizing depends on the tier at which a task fails and on how many queries one repository snapshot will serve. Neither scalar Pass@1 nor latency figures alone can show this.
Core Claim
We make three linked, falsifiable claims. First, context saturation in repository code does not damage all abilities equally. As distractor code is added, failures will concentrate at the relational tier (following which callee's return value feeds the target) before they show up as recall failures. Second, test-time training on repository chunks, in the style of ETT, will raise recall accuracy at long effective context. It will also raise the rate at which the model calls symbols that do not exist when a needed definition is missing. Our reasoning is that weight-stored memory has no provenance, whereas in-context text does. Third, small general-purpose instruction-tuned models will beat code-specialized models of similar size mainly on the relational tier and on the unsolvable controls, not on recall. This would extend to code the pattern that domain tuning inflates agreement with false premises (MedREQAL).
Where This Sits
Three strands of prior work motivate the proposal. RepoDebug shows that Claude 3.5 Sonnet's bug-identification accuracy falls from 51.48% to 43.07% as code grows past 500 tokens. SWE-bench shows that telling the model which files to edit raises Claude 2 from 1.96% to 4.8% of issues resolved. Both report single scalar scores and cannot say which ability failed. ToolBeHonest supplies the method we borrow. It tests models at three levels: whether a task is solvable, where the bottleneck is, and what the missing tool would look like. It finds that models scoring 90+ on solvable tasks collapse on unsolvable ones. We carry that design into repositories, where the "missing tool" is a deleted function, and DevEval already notes that models hallucinate functions when cross-file dependencies are omitted.
Two lines of work supply the remedies we compare, and neither diagnoses its effect by tier. Hierarchical Context Pruning parses the repository into a syntax tree and keeps function bodies only for directly imported files and headers only for distant files. It cuts roughly 50k tokens to 8k. It was tested only on code-specialized models and only on completion. DyCoder has an LLM walk call chains on demand to build a partial dependency graph. It pays for this with sequential model calls. ETT is the memorizing alternative. It fine-tunes only the feed-forward up-projection layers on overlapping 512-token chunks. It reports LongBench gains of up to 30% and a 32x window extension, at the cost of a long delay before the first token. It treats input as a flat stream and was run on GPT-Large and Phi-2 on prose benchmarks, never on code and never under unsolvable controls. Our proposal differs from all three because it creates no new remedy. It asks which failure each remedy fixes, and at what latency.
TAGS and ∇-Reasoner constrain the design. TAGS shows that test-time generalist–specialist collaboration beats fine-tuned medical models, but only on short multiple-choice questions, and it concedes that the multi-call latency is high. ∇-Reasoner shows that gradient search can beat Best-of-N at fewer model calls, but only on isolated math problems. We take from them that latency must be reported alongside accuracy, and that discrete sampling is a real competitor that cannot be waved away.
Research Program
The first component is a tiered probe set built from DevEval and the Python and Java portions of CrossCodeEval. Both come with dependency ground truth from static analysis. For each target function we generate three tiers:
- Recall: reproduce the exact signature of a named callee defined in another file.
- Relational: name the chain of definitions through which a given argument's type is resolved, scored against the static call graph.
- Composition: the benchmark's own generation task, scored by Pass@1 on DevEval and exact match on CrossCodeEval.
Tiers are scored for the same instance. This lets us measure non-monotonic cases, such as composition succeeding after recall fails, which flag guessing or contamination. Effective context length is controlled by padding the true dependencies with real distractor files from the same repository, at budgets of 2k, 8k, and 32k tokens.
The second component is the unsolvable control. For a matched subset, we delete one required dependency's definition from the repository snapshot. The deletion applies to the prompt, the retrieval corpus, and the test-time training corpus alike. Instruction-following models are told to emit a marker naming the missing symbol. For completion-only models we detect, with a static checker, whether the generated code calls a symbol undefined in the snapshot. We report three things: the rate of fabricated calls, the rate of correct abstention, and, mirroring ToolBeHonest's third level, whether the model's proposed signature for the missing function matches the deleted one. Pretraining may have exposed models to the original repositories, so we stratify results by repository creation date. CrossCodeEval repositories date from March–June 2023.
The third component is a latency-matched comparison of remedies. The same open-weight models are run under four regimes:
- current-file only;
- HCP-style hierarchical pruning;
- DyCoder-style LLM dependency navigation;
- ETT-style up-projection training on repository chunks.
Test-time training is run in two accounting modes. In per-query mode, training happens fresh for each query. In per-snapshot mode, one training run is shared across all queries against that repository version. For per-snapshot mode we report amortized latency,
$$L(q) = \frac{T_{\text{train}}}{q} + T_{\text{infer}},$$
where $T_{\text{train}}$ is the wall-clock training time on a snapshot, $T_{\text{infer}}$ is per-query generation time, and $q$ is the number of queries served before the snapshot changes. We then locate the break-even $q$ at which memorizing matches reading on each tier.
Models fall into two matched groups at roughly 7B parameters. The code-specialized group is DeepSeek-Coder, StarCoder2, and CodeGemma, the models HCP used. The general group is Qwen2.5-7B-Instruct and Llama-3-8B-Instruct. Frontier closed models enter only the inference-only regimes.
Baselines
The naive reference point is current-file-only generation. CrossCodeEval reports StarCoder at only 8.8% exact match on Python in this setting. Every remedy must clear this floor on every tier. Clearing it only on composition would suggest guessing.
The strongest reading-based competitor is Hierarchical Context Pruning, which reports CodeGemma-7B exact match rising from 36.69% to 41.21% on CrossCodeEval. It is the strongest competitor because it is cheap, single-pass, and already wins on five of six models. We reimplement it with its 8k budget on identical snapshots and the same decoding settings. Memorization "beats" it only if it improves a specific tier at equal or lower amortized latency for a stated $q$, and does so without raising the fabricated-call rate on unsolvable controls.
The strongest navigation-based competitor is DyCoder, which reports a 59.73% relative Pass@1 improvement on DevEval and 7.4x faster retrieval than RepoScope. No absolute Pass@1 for DyCoder was reported on Lacuna. The only absolute DevEval anchor we have is gpt-3.5-turbo-1106 going from 18.03 to 44.83 with local plus oracle context (DevEval), which we treat as an upper reference. DyCoder is given the same per-query wall-clock budget as test-time training, with its navigator set to the same base model.
Best-of-N sampling with self-generated tests is the discrete test-time alternative. Its accuracy-versus-calls curve is compared at matched GPU-seconds. ∇-Reasoner's MATH-500 result (71.2% to 80.4% for Qwen-2.5-7B-Instruct, with up to 40.2% fewer calls than Best-of-N) shows that sampling can be beaten on math, but no such result exists for repository code.
Two ablations isolate the key ingredient. In the first, test-time training runs on chunks from an unrelated repository of equal length. This separates gains from repository content from gains caused merely by perturbing the weights. In the second, test-time training is combined with HCP headers in context. If provenance is what prevents fabrication, the hybrid should keep the recall gain and bring the fabrication rate back down.
Why not just…?
Why not just report Pass@1 and skip the tiers? Pass@1 is the outcome we care about, but it cannot test our main claim. That claim is that memorization helps recall while harming solvability judgment, and these two effects can cancel in a single score. SWE-bench shows the value of separating localization from repair. ToolBeHonest shows that a 90+ score on solvable tasks hides collapse on unsolvable ones. If tier scores turn out to be explained almost entirely by Pass@1 (we will report the correlation), the tiers are redundant and we will say so.
Why not just put the whole repository into a long-context model? For the frontier generalists this is a genuine competitor. We include it as a regime at the 32k budget. Existing evidence is against it: HCP's 50k-token prompts underperform its 8k pruned prompts, and RepoDebug shows degradation already beyond 500 tokens. If long-context ingestion matches pruning on the relational tier at 32k, the saturation hypothesis fails at that scale.
Why not just fine-tune on each repository offline? That is exactly what per-snapshot test-time training becomes when $q$ is large, so we do not argue around it. The amortized-latency analysis is the experiment that answers it. If fabrication on unsolvable controls rises with training, offline per-repository tuning inherits the same problem, as MedREQAL found for medical tuning. If fabrication does not rise, offline tuning is simply the recommended configuration for stable repositories.
Why not just sample many completions and keep the one that passes self-generated tests? Self-generated tests are written by the same model that fabricates. A completion calling a nonexistent helper may pass a test that mocks it. We measure this directly by reporting fabrication rates among test-passing Best-of-N outputs on the unsolvable subset. If Best-of-N matches every remedy at equal compute on all tiers, it wins and the proposal's practical recommendation collapses to "sample more."
Decisive Checks
Three outcomes would settle the proposal.
The first check is the recall result. At the 32k distractor budget, per-snapshot test-time training must improve recall accuracy over HCP at equal amortized latency for some realistic $q$. If it never does, memorization has no niche in code.
The second check is the fabrication result. Fabricated-call rate on unsolvable controls should be higher after test-time training than under HCP, and should fall again in the hybrid ablation. If the rate does not rise, the provenance hypothesis is wrong and memorization is safe. If the rate rises in the unrelated-repository ablation too, the cause is weight perturbation rather than content.
The third check is the specialist result. The general–specialist gap at equal size should be concentrated in the relational tier and the unsolvable controls rather than in recall. A uniform gap would mean the generalist advantage in code is broad competence, not judgment about what exists.
Together, these checks yield a concrete deployment rule: read when tasks fail relationally or when the repository changes often, and memorize only when failures are recall failures and the snapshot serves enough queries to pay back the training time.
References
- ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time
- MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering
- RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- DevEval: Evaluating Code Generation in Practical Software Projects
- Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs
- Effective and Efficient Context Retrieval via Partial Dependency Graph for Repository-Level Code Generation
- TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
- $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion