When a code model completes a line inside a large software project, it is usually shown snippets pulled from other files. Some of those snippets help. Some are harmful because they look right without being right: a method with the same name on a different class, an older helper with a different argument order, or a near-duplicate utility in a sibling module. We think a large share of the "harmful retrieved context" reported in repository-level completion comes from these name collisions. If so, the model cannot fix the problem by attending harder to the snippets, because the snippets themselves do not say which binding is in scope. That information lives in two places: the local code (what type the receiver variable has) and the project's import and signature structure (which definition that name actually resolves to).
The proposal is to give the model a compact symbolic map of the project, listing what this file imports and the signature each imported name resolves to. The model is then trained on counterfactual examples. Each pairs the true cross-file dependency with a verified look-alike distractor, built so that only the map plus the local code can tell them apart. Our central bet is an interaction effect. Distractor training without the map gives the model nothing to disambiguate with. The map without distractor training gets ignored, because on ordinary crawled code a lexical match is usually good enough. Only the two together should teach the model to resolve bindings rather than copy whatever matches on the surface.
Core Claim
Prior work establishes that retrieved cross-file context is often useless or harmful. Repoformer fine-tunes StarCoderBase to decide whether to retrieve at all, and reports that up to 80% of retrieval events are unhelpful or harmful. CODEFILTER labels each retrieved chunk as positive, neutral, or negative by how much it changes the log-likelihood of the true code, then trains a filter to drop the bad chunks. Both treat a harmful chunk as something to avoid. Neither asks why it is harmful, and neither gives the model a way to recognize it from structure.
RepoBench adds a tension: it reports that randomly chosen repository snippets beat no cross-file context at all. Random context can therefore act as a useful project-level prior, while CODEFILTER finds that specific retrieved chunks hurt.
Our claim resolves this tension with a mechanism. Context mostly hurts when it is a near miss on identifiers and signatures. Retrievers rank near misses highly precisely because they match lexically, which is why retrieved context can be worse than random context. The fix is to train the model to fuse a global symbol map with local binding cues, not to remove context.
If the claim holds, three things should follow. A model trained this way should keep most of the benefit of noisy retrieval and lose most of its harm. It should do so without the extra predictor that CODEFILTER needs at inference time. And the gains should carry over to the naturally occurring negative-chunk instances that CODEFILTER identified.
How the Four Directions Fit
Cross-file dependency modeling is the application. Contextual noise sensitivity is the phenomenon we explain and target. Global-local fusion is the mechanism, and LLM-based data augmentation is one way to manufacture training signal, which we test against a cheaper deterministic way rather than assume it is needed.
The fusion idea borrows from TopicBERT. TopicBERT concatenates a cheap global document representation with a truncated local encoding and recovers nearly full-length classification quality. Its global signal is a bag-of-words topic vector, which discards exactly the relational, symbolic information code needs. CoCoMIC compresses imported entities into learned summary vectors that the model attends to jointly with in-file tokens, but it has no notion of an entity being a misleading match.
Our global signal is different from both. It is a short, plain-text "symbol skeleton": one line per imported name, giving its resolved module path, its kind, and its signature. It is built with tree-sitter, the same parser RepoGraph uses for its line-level graphs. We deliberately keep the skeleton as ordinary prompt text, so any gain comes from training signal and not from a new architecture.
The augmentation side takes its structure from CLIPPER. CLIPPER shows that conditioning an LLM generator on compressed structure, rather than raw long text, cut synthetic-claim errors from 73.1% to 16.7% on narrative data. We condition generation on the symbol skeleton, not on whole files.
The constraint comes from CACP, which shows that code models drop 10–25% under variable renaming and up to 34% under semantic-preserving mutations. That work covers only isolated functions. We move its counterfactual logic across file boundaries, and we take its results as a warning that surface-level perturbations alone may teach the wrong lesson.
Research Program
Building collision distractors. For each completion target in a training repository, static analysis identifies the cross-file definition the true completion depends on. We then construct distractors in two arms.
The deterministic arm edits the abstract syntax tree of the true definition. It can permute or rename parameters, move the definition onto a sibling class, or produce a deprecated variant with a changed return type. It also transplants same-named definitions from other repositories.
The LLM arm prompts a code model with the skeleton and the true definition. It asks for a plausible sibling module containing a same-named but different API, with a coherent body and docstring, which is the kind of tempting near miss that mechanical edits produce poorly.
Every distractor in both arms passes one verification step before use, adapted from the static-analysis check in CrossCodeEval. The distractor must parse. The true completion must still resolve against the real repository. A completion written against the distractor's API must fail static resolution or type checking against the real repository. This check guarantees that every distractor is truly wrong and never merely an alternative valid answer. It also addresses the known tendency of synthetic code to hallucinate APIs.
Training. Each training prompt contains the skeleton, the in-file prefix and suffix in fill-in-the-middle format, and a mix of retrieved chunks. The mix includes the true chunk, ordinary retrieved neighbors, and a verified distractor that is inserted in some examples and withheld in others. The loss is the standard next-token loss on the true completion. We add no auxiliary head, so the model must learn on its own to use the skeleton to discount the distractor.
The base corpus is the Stack-derived instances released with CODEFILTER. This lets us train under matched data, and it lets us reuse their positive, neutral, and negative chunk labels for analysis.
Diagnosing the random-versus-retrieved tension. We vary the type of distractor at test time while holding prompt length fixed. The conditions are random repository snippets, lexically similar but non-colliding snippets, and verified collision distractors. Our hypothesis predicts that random snippets are harmless or mildly helpful, consistent with RepoBench. It also predicts that harm concentrates almost entirely in collision distractors, and that most CODEFILTER-negative chunks found in the wild are collisions under our static definition.
Baselines
The naive reference points are in-file-only completion and always-on retrieval with the same retriever and chunk budget, using no skeleton and no distractor training. CrossCodeEval reports in-file StarCoder at 8.8% exact match on Python, with retrieval often doubling or quadrupling exact match. We reproduce both points on our backbone instead of relying on those figures.
The strongest competitor for handling harmful context is CODEFILTER, a trained filter that removes neutral and negative chunks before generation. It is the closest existing answer to our exact question. It reports an average 3% exact-match gain over baseline retrieval-augmented generation on RepoEval and more than 10% on instances with negative chunks. We train its filter on the same Stack-derived data, apply it to the same retrieved pool, and generate with the same backbone that we fine-tune. We charge its extra filter forward pass to its latency.
For structural retrieval, the competitor is DRACO, which traces dataflow from the cursor to fetch only provably related definitions. It reports a 3.43% exact-match gain and a 3.27% identifier-F1 gain over RepoCoder, about 40ms versus 4000ms for prompt construction. We run it on the Python portions of CrossCodeEval and RepoEval.
Repoformer is a selective-retrieval competitor. We found no reported exact-match result on Lacuna for Repoformer, so we train its retrieve-or-not token on the same backbone and data. RepoCoder's iterative retrieval serves as the ordinary retrieval pipeline that both we and the filters sit on top of.
The ablations that isolate the key ingredient form a two-by-two grid: with or without the skeleton in the prompt, crossed with with or without distractor training. Inside the distractor-trained cells we compare the deterministic and LLM arms at equal example counts. All fine-tuned systems get the same number of training tokens, steps, and hyperparameter sweep size.
Beating CODEFILTER means matching or exceeding its exact match on RepoEval and CrossCodeEval overall, beating it on the negative-chunk subset and on the collision test set, and doing so with no inference-time filter.
Why not just…?
Why not just filter out bad chunks at inference, as CODEFILTER does? A filter must itself tell a collision from the true dependency. Without the skeleton it faces the same ambiguity the generator does, and it adds a forward pass per chunk. Filtering is still a real threat, because its 10%-plus gain on negative-chunk instances is exactly the regime we target. We run it head-to-head and also stacked with our model. If the stacked system gains nothing over the filter alone, our training adds nothing.
Why not just retrieve precisely with dataflow analysis, so distractors never enter the prompt? DRACO's numbers show this works where static resolution succeeds. The file notes that it fails under dynamic typing and indirect exports, and that it is Python-only. Production systems also add lexically similar usage examples that dataflow would not fetch. We report DRACO separately on the subset where its resolver returns the true definition and on the subset where it does not. If our method only helps where DRACO already fails, that is a narrower but still honest result.
Why not just drop every chunk whose file is not imported? This rule is cheap, and it removes some transplanted distractors. It also removes the usage examples from sibling files that iterative retrieval in RepoCoder relies on. It does nothing against same-named members on a different class inside an imported module. We include it as a baseline. If it matches our model, the skeleton's value reduces to this rule and the training contribution is void.
Why not just use deterministic edits to the syntax tree instead of an LLM to make distractors? Plausibly this works as well. That is why the LLM arm is a tested variant and not part of the claim. If the deterministic arm matches it, we recommend the cheaper generator, and the interaction claim stands without the LLM.
Why not just put the whole repository in a long-context model? LongCodeU reports that inter-file relation accuracy falls from about 50% at 8K tokens to near 0% at 64K–128K. More context brings in more collisions, not fewer.
Proposed Experiments and Decisive Checks
We fine-tune a StarCoderBase-class backbone, the model family Repoformer used, at two sizes. We evaluate on CrossCodeEval in all four languages, RepoEval at line, API, and function granularity (with unit-test pass rate for functions), and CrossCodeLongEval. We also evaluate on a held-out collision test set built from repositories disjoint from training. Its distractors come from a generator family not used in training, so that the model cannot win by spotting a specific generator's style. Metrics are exact match, edit similarity, and identifier F1, plus a distractor capture rate: the fraction of errors whose predicted identifier or signature matches the distractor's.
The first decisive check is the interaction. Skeleton plus distractor training must beat each single-ingredient cell on the collision set by more than the sum of their individual gains over the base.
The second is reliance. Removing the skeleton at test time should hurt the jointly trained model on collisions much more than it hurts the skeleton-only model, which shows the model actually reads the map.
The third is transfer. On RepoEval instances that CODEFILTER's labels mark as containing negative chunks, our model must narrow or close the gap to the filter without running a filter.
The fourth is no regression. On instances where retrieval is unnecessary, which Repoformer reports are the majority, exact match must not fall below always-on retrieval.
If harm turns out not to concentrate in collision distractors in the diagnostic study, the core explanation is wrong. We would report that directly as the resolution of the RepoBench–CODEFILTER tension.
References
- Repoformer: Selective Retrieval for Repository-Level Code Completion
- Impact-driven Context Filtering For Cross-file Code Completion
- RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems
- TopicBERT for Energy Efficient Document Classification
- CoCoMIC: Code Completion by Jointly Modeling In-file and Cross-file Context
- RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph
- CLIPPER: Compression enables long-context synthetic data generation
- Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates
- CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation
- Benchmarking Long-Context Language Models on Long Code Understanding