Occam Attribution: Explaining Generated Code with the Fewest Necessary Sources

When a code model produces a function that looks like someone's proprietary work, the practical question is whether this output owes something specific to a particular source file, and if so, to which files and how few. The tools in use today answer a cruder question: whether the output shares text with something in the training corpus. That cruder question fails in two directions. It flags generic boilerplate that any competent model would write. It also misses refactored or translated copies whose surface text has changed. We propose to treat attribution as parsimonious hypothesis selection, the classical AI practice of trying the simplest explanation first and adding complexity only when the evidence forces it. The simplest hypothesis is "this code is generic and needs no source." The next is "one source explains it." Then come two sources, and so on. Each step up the ladder must be earned by a calibrated statistical test.

The core measurement is easy to state. A reference code model, which never saw the candidate source, scores how predictable the output is. It scores the output once on its own and once with a candidate source shown in its context. Boilerplate gains almost nothing from seeing any particular source, because the reference model already predicts it. A refactored copy of a specific algorithm gains a great deal, even when few tokens match, because the source tells the model what logic comes next. Wrapping this gain in a null distribution, an ascending ladder of explanation sizes, and a merging step that prefers one source covering many fragments over many sources covering one each turns a similarity score into a minimal, testable attribution claim.

Core Claim

Attribution of generated code should report the smallest set of sources whose inclusion is statistically necessary to explain the output, measured by predictive gain over a source-free reference model. On refactored clones and on outputs stitched from several parents, this procedure should beat both exact-match tracing and pairwise clone detection. On boilerplate it should produce far fewer false attributions. The claim is falsifiable. If corpus-frequency-weighted n-gram matching does as well on refactored and multi-parent cases, the idea is not worth pursuing.

Where This Sits

The motivating evidence comes from the code-IP side. StarCoder ships an attribution tool that matches generated code against The Stack by substring. Its own limitations are that it cannot tell boilerplate from protected logic and that it misses semantic translations and refactorings. We target exactly those two failures. Oreo detects "twilight zone" clones (below 70% syntactic similarity) with a staged pipeline: cheap size and API-usage filters first, then a Siamese network. We reuse its staging idea for candidate retrieval. Oreo, however, is a pairwise classifier. It does not say whether a clone is meaningful rather than generic, and it cannot resolve derivative works with several parents.

The statistical form of the test comes from Proving membership in LLM pretraining data via data watermarks. That paper frames provenance as rejecting a null hypothesis, using loss on watermarked text against loss on matched random text, which yields bounded false-detection rates. We keep the null-hypothesis structure but drop the requirement that owners plant watermarks before release, so the method works on unprepared, historical code.

CoDEMI constrains our design. It asks a related question, whether a file was in the training set, using token ranks and shadow models. Its accuracy falls from 0.842 on an LSTM to 0.730 on CodeGPT and drops significantly on large models such as StarCoder, because large models generalize rather than memorize. We take this as a warning not to rely on the target model's memorization signal. Our test instead measures the dependence of the output on a source, using a separate reference model, and does not ask whether the target memorized anything.

The mechanism comes from the parsimony literature. Automated Model Selection for Simulation and MECHEM show that searching a ladder of increasing complexity is tractable and avoids overfitting where top-down pruning is not. On the Mechanization of Abductive Logic supplies the merging rule: when several observations trace to one cause, collapse them into a single hypothesis. In our setting the observations are the flagged code fragments. The noise problem documented for these ladder methods, where valid simple hypotheses get rejected on noisy data, is real here too. We address it by replacing exact-fit refutation with calibrated tests.

The nearest work at the intersection is retrieval-based IP isolation. SILO keeps risky data in a deletable datastore. CPR bounds the information any single retrieved image contributes to a generation. Both control how sources enter a generator. Neither explains after the fact which sources an existing output depends on, which is our question.

Research Program

We start from a generated snippet $y$ and a corpus of candidate sources. Stage one retrieves a few hundred candidates per snippet with cheap filters in the Oreo style: size similarity and overlap in API calls, followed by embedding search.

Stage two scores each candidate $s$ by its conditioning gain:

$$G(s, y) = \log p_R(y \mid s) - \log p_R(y),$$

where $p_R$ is a reference code model trained on a permissive corpus that excludes the candidate pool. $p_R(y \mid s)$ is its likelihood for $y$ when $s$ is placed in its context, and $p_R(y)$ is its likelihood with neutral context. We also keep per-token gains, so we can see which fragments of $y$ each source explains.

The null distribution for $G$ comes from matched decoy sources: files of similar length, language, and API profile that provably could not have influenced $y$, such as Github-Python files created after the generator's training cutoff. Each candidate therefore receives an empirical p-value.

The ladder then runs in order. The generic hypothesis, that no source is needed, stands unless some candidate's p-value survives a correction for the number of candidates tested. If a single source survives, we ask whether the fragments of $y$ that are poorly predicted by $p_R$ are now covered. Only if a significant uncovered region remains do we test a second source, conditioning on the first. The merging step runs before any escalation: if one source's per-token gains already cover the fragments that two separate sources explain, the single-source hypothesis wins.

The output is a short list of sources with p-values and fragment-level coverage maps. A lawyer or license-compliance engineer could read and contest this report. It is not a similarity ranking.

Baselines

The naive reference point is reporting nothing, the always-generic hypothesis. It has zero false attributions and zero recall, and it shows how much each method gains over abstaining.

The first substantive baseline is exact-match tracing in the style of StarCoder's attribution tool: longest common substring against the candidate pool, thresholded. No reported result found on Lacuna for its attribution accuracy.

The baseline we consider the strongest skeptical competitor is corpus-frequency-weighted n-gram matching. It scores overlap with each source but down-weights n-grams that are common across the corpus. This directly targets the boilerplate problem with no model at all. All three surface-matching baselines share our candidate retrieval stage and are tuned on the same development split, with the same search budget over n-gram length and threshold.

The strongest existing learned method is a clone detector used as an attributor: Oreo, or a CodeBERT clone classifier. The latter reports 96.5 F1 on BigCloneBench in CodeXGLUE. Oreo reports 30% recall on Moderately Type-3 clones, against 5% for SourcererCC, at 89.5% precision. We attribute to every candidate whose clone score passes a threshold tuned for the same false-attribution rate as our method on the development set.

Beating these methods means higher file-level attribution recall at matched false-attribution rate on refactored and multi-parent cases, together with a lower false-attribution rate on boilerplate at matched recall.

Three ablations isolate the key ingredients. In the first, "no ladder," every source passing the threshold is reported, which tests whether parsimony itself lowers false attributions. In the second, "no null," raw gain is thresholded instead of calibrated p-values. In the third, "surface gain," the likelihood gain is replaced by n-gram overlap inside the same ladder. This shows whether predictive dependence, rather than the search structure, carries the result.

Why not just…?

Why not just down-weight common n-grams? This is the real threat. On verbatim and near-verbatim copies we expect it to match our method, and we will say so. Its weakness should appear in two places. First, refactored clones keep the logic but rename and reorder, leaving few distinctive n-grams. Second, protected code that was itself widely duplicated looks "common" and gets suppressed. Our benchmark includes both slices explicitly. If frequency weighting matches our method there, the predictive-gain idea is unnecessary and we will report that.

Why not just use a strong clone detector? Clone detectors answer whether two pieces of code are similar. They do not answer whether the output needed this source rather than generic knowledge, so they should over-attribute on idiomatic code that many files share. They also give no principled way to choose among several matching parents. Oreo's own scope, pairwise and with no multi-parent reasoning, reflects this. The boilerplate slice and the multi-parent slice test whether that reasoning matters in practice.

Why not just run membership inference on the target model? Membership is a different question: a file can be in the training set without shaping a given output. CoDEMI also shows that rank-based membership signals weaken sharply as models grow. Because our test uses a separate reference model and measures dependence of the output on the source, it should not degrade as the target model scales. We check this by running the same attribution on outputs from small and large generators.

Proposed Experiments

We build an attribution benchmark with known ground truth in four slices.

The verbatim slice plants protected Java and Python files inside generated contexts. The refactored slice draws BigCloneBench Type-3 and Type-4 pairs: one member goes into the candidate pool and the other serves as the "output," with the pool also holding IJaDataset distractors. The multi-parent slice splices functions from two or three pool files and then refactors them. The boilerplate slice samples generic outputs from a model whose training data excludes the pool, so that no source should be credited.

The reference model is trained on a permissive subset of The Stack or on the Open License Corpus, with all pool files removed. Decoys come from post-cutoff Github-Python. Metrics are file-level precision and recall, false-attribution rate on the boilerplate slice, calibration of p-values on decoy-only runs, and the size of the reported source set compared with the true number of parents.

The decisive check uses counterfactual ground truth that does not depend on our own scoring. On Py150, we fine-tune small CodeGPT-scale generators with and without specific protected files. A source "truly matters" for an output if removing it from training measurably lowers that output's probability under the retrained generator. This is affordable only at small scale, which is why we do it there.

The proposal succeeds if three things hold. Our attributions should agree with this counterfactual signal better than frequency-weighted n-grams and clone detectors do on the refactored and multi-parent slices. Our false-attribution rate on boilerplate should stay near the nominal test level. The ladder ablation should show that escalating only when forced cuts spurious extra sources without losing true parents.