Randomized Program Serializations for Auditing Code-Execution Benchmarks

Chemists learned years ago that one molecule can be written as many different strings. The choice of starting atom and traversal order is arbitrary. A model trained on only one spelling per molecule learns the spelling rather than the chemistry, and it fails when the notation changes. We propose to use the same lesson to answer a question that code-reasoning evaluation cannot currently answer: when a language model correctly predicts what a benchmark program prints, is it simulating the program or recalling a memorized answer? Programs, like molecules, admit many equivalent serializations. Independent statements can be reordered, variables renamed, and branches flipped with negated conditions, all without changing what the program computes. Because programs can be run, the correct answer for every variant comes free from an interpreter, with no human or teacher model needed.

The core contribution is an audit for execution benchmarks such as CRUXEval, CoCoNUT and CodeSense. It separates two failure modes that existing mutation studies conflate. The first is sensitivity to notation: the model is thrown off by unusual code. The second is memorization: the model has stored the benchmark's specific input–output pair. The audit does this with two ingredients. A matched control set of never-published programs measures how much accuracy falls under re-serialization when memorization is impossible. A "near-miss" probe makes a one-token edit that changes the program's true output and checks whether the model still returns the original benchmark answer. We validate the audit by deliberately contaminating open models with known halves of each benchmark and checking that it recovers which half leaked.

Core Claim

A drop in accuracy under equivalent rewrites is not evidence of contamination by itself. Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates mutates reference code with if-else flips, line swaps, def-use breaks and renaming. It finds that renaming alone costs 10–25% accuracy and some mutations cost up to 34%. Part of that drop reflects broken code "naturalness" and tokenization, not recall. The molecular analogue makes the same point: Chemical Language Models Have Problems with Chemistry shows MolT5's caption METEOR falling from 0.583 to 0.329 when explicit hydrogens are added, a pure notation shift. A contamination signal must therefore be an excess drop on benchmark items relative to matched items the model cannot have seen. For an execution task, a much sharper signal is also available: the stale-answer rate. This is the fraction of near-miss variants, whose true output has changed, on which the model outputs the original program's output. A model that simulates the code has no reason to produce the old answer. A model that recalls it does.

Let $A_B(\cdot)$ and $A_C(\cdot)$ be accuracy on benchmark items and on matched control items. Let "orig" denote the published serialization and "equiv" a randomized equivalent serialization. The excess-drop statistic is

$$\Delta = \big[A_B(\text{orig}) - A_B(\text{equiv})\big] - \big[A_C(\text{orig}) - A_C(\text{equiv})\big],$$

which is near zero for a clean model and positive when benchmark scores are propped up by memorized surface forms. Both $\Delta$ and the stale-answer rate can also be computed per item, which gives item-level contamination scores.

Relation to Prior Work

Two lines of work motivate this proposal by documenting the gap. Execution benchmarks show that models generate code much better than they trace it. CRUXEval tests output and input prediction on 800 short Python functions. CoCoNUT reports that a model passing 80% of HumanEval generation can trace only about 40% of the same programs. CodeSense finds statement-level state prediction under 30% in C. None of these papers asks whether the scores they do report are inflated by leakage. The question is pressing for CoCoNUT's HumanEval-Trace, whose programs are canonical HumanEval solutions. Quantifying Contamination shows such solutions widely present in pretraining data: 20.8% of MBPP appears verbatim in StarCoderData.

A second line supplies methods we reuse but constrains how we use them. DyCodeEval rewrites the natural-language problem story with a multi-agent Claude pipeline and scores consistency across variants. Its controlled experiment, fine-tuning Llama-3.2-1B on leaked data, is the validation design we adopt. Its mutations, however, target generation prompts, not the code a model must execute, and they cost teacher-model calls. CAP detects contamination through a consistency ratio under meaning-preserving perturbations of financial QA. We borrow the consistency idea but gain something CAP lacks: an interpreter that labels every variant exactly. Concerned with Data Contamination? constrains the design in two ways. It shows that perplexity cannot separate seen from unseen code. It also finds that syntactic refactoring sometimes raised completion accuracy, which contradicts the large if-else-flip effects in the CACP study. Our matched control directly measures each transformation's notation cost, so it lets us tell which transformations are informative probes and which are confounded. Rethinking the effects of data contamination in Code Intelligence found that only paired input–output leakage inflates scores and that later task fine-tuning erases the effect. For execution benchmarks the paired case is precisely the one that matters, and whether later tuning erases it is one of our tests.

From chemistry we take a specific mechanism, not a loose analogy. MolPMoFiT trains on randomized SMILES and averages predictions over several random spellings at test time. It reaches 0.565 RMSE on Lipophilicity against a prior 0.589. Here, randomized serializations serve first as a probe and second as a candidate remedy.

Research Program

We will build a deterministic transformation library over Python abstract syntax trees, the parsed tree form of a program. It enumerates equivalent serializations in the same spirit as randomized SMILES enumerates spanning-tree traversals. It covers topological reorderings of statements with no data dependence between them, consistent renaming of local identifiers to both natural and unnatural names, if-else flips with negated guards, and reordering of helper-function definitions. Every variant is executed to confirm that it produces the original output. For CoCoNUT, variants are re-instrumented to obtain the new line-number trace. A separate near-miss generator applies single-token edits, such as changing a constant, a comparison operator or a slice bound, and keeps only those whose executed output differs from the original.

The control set is produced by re-running CRUXEval's generate-and-filter procedure now, with a fresh seed set, and withholding the results from publication. Items are matched to benchmark items on line count, operation types used and trace length. Difficulty is matched with a reference model trained only on data that predates the benchmark's release. This avoids matching on the very accuracy that contamination inflates.

The audit then computes $\Delta$, the stale-answer rate and per-item scores for a range of open and API models, separately for output prediction, input prediction and line tracing. The transformation families are reported separately so that the conflicting refactoring findings between Cao et al. and the CACP study can be resolved per family. Last, a smaller follow-up asks the chemistry question directly. If a 1B–7B open model is fine-tuned on execution traces with randomized-serialization augmentation, does its benchmark accuracy become invariant to serialization? And does deliberately injected leakage then inflate its static score less than it inflates a model trained on canonical code?

Baselines

The naive reference point is the static benchmark score on published serializations. Examples are CRUXEval's reported GPT-4 chain-of-thought accuracy of 81.9% on output prediction and 75.0% on input prediction, and Code Llama 34B's 46% and 50% (CRUXEval). The naive reference treats every score as clean, and any detector must beat chance at identifying leaked items where this reference cannot.

The strongest existing competitor is a CACP-style mutation-effect score transplanted onto execution tasks. It applies the four mutations of Hooda et al. to CRUXEval and CoCoNUT programs and treats per-item accuracy drop as the contamination score. This is the obvious thing a reviewer would do. The reported reference values are mutation effects up to 34% on generation and roughly 15% for if-else flips with GPT-4 on CodeContests. No reported result on execution tasks was found on Lacuna. A CAP-style consistency ratio over the same variants is a second strong competitor. No reported PCR values were found on Lacuna. A perplexity membership-inference baseline is included because practitioners still use it, though Cao et al. report mean perplexity barely moving (3.37 to 5.17) between pre- and post-cutoff code.

All detectors run on the identical variant pool, the same models, the same decoding settings and the same number of variants per item. No detector gets tuned thresholds on the evaluation half. Thresholds are chosen on a held-out contamination split for all methods alike.

Two ablations isolate the key ingredients. The first removes the control correction, which reduces $\Delta$ to a raw drop and is essentially the CACP baseline. The second removes the near-miss probe. Beating the strongest baseline means a statistically clear improvement in AUROC for identifying which half of a benchmark was leaked in the controlled-contamination experiment. It also means a markedly lower false-positive rate on control items, where the raw-drop baseline should flag naturally brittle but clean programs.

Why not just…?

Why not just apply the CACP mutations to CRUXEval and report the drop? Because the drop mixes notation sensitivity with recall. The CACP paper itself attributes much of its renaming effect to broken naturalness and tokenization, and Cao et al. find refactoring effects of the opposite sign. This is a real threat only if notation cost turns out to be negligible on execution tasks. The no-control ablation settles it: if raw drop matches $\Delta$ in leak-detection AUROC and false-positive rate, the control adds nothing and we will say so.

Why not just evaluate on fresh, post-cutoff programs? Fresh programs give a clean number going forward but say nothing about scores already published on CRUXEval or CoCoNUT, which is what the field cites. Temporal cutoffs also leak: Cao et al. observe StarCoder improving from about 30% to about 34% on later code because recent repositories mirror model output. We do use fresh programs, but as a control that calibrates the audit, not as a replacement benchmark.

Why not just rewrite items with an LLM, as DyCodeEval does? For execution tasks the label must change exactly when the code changes. An LLM rewrite risks silent semantic drift and costs teacher-model calls, and DyCodeEval needed human verification to reach 95% consistency. Abstract-syntax-tree rewrites checked by an interpreter are exact and essentially free. As a sanity check, we will run DyCodeEval-style rewriting on a subset to test whether it adds detection power beyond our transformations.

Why not just give the model an interpreter? That would make the model more useful but would not measure anything about the model. Input prediction also cannot be solved by forward execution, and it is the task where CodeSense reports about 12% for the best model.

Proposed Experiments

The decisive validation follows DyCodeEval's design. DyCodeEval reports that leakage fine-tuning lifted Llama-3.2-1B's static score from about 20% to about 80% while its dynamic score stayed near 13%. We fine-tune open code models of 1B to 7B parameters on a random half of CRUXEval, with outputs, at one, five and twenty exposures, and do the same separately for CoCoNUT HumanEval-Trace. Each detector must then identify the leaked half. We expect the stale-answer rate to rise steeply with exposure count, $\Delta$ to rise moderately, and raw drop to show high false positives on control items. If near-miss stale answers do not rise even at twenty exposures, the core claim fails. We then add instruction tuning on unrelated code after leakage to test whether Yang et al.'s finding, that fine-tuning resets contamination, holds for execution tasks, and we report which statistic survives.

The audit itself applies the validated statistics to publicly available models on CRUXEval, CoCoNUT HumanEval-Trace and the Python portion of CodeSense, separately for output prediction, input prediction and tracing. We predict a larger contamination signal on HumanEval-Trace than on CRUXEval, given HumanEval's documented leakage. Finding no difference would itself be informative.

The augmentation follow-up compares three fine-tuning conditions on equal compute: canonical-only training, randomized-serialization training, and the canonical-trained model with MolPMoFiT-style voting over serializations at test time. Each is measured by accuracy on equivalent variants of control items and by how much injected leakage inflates its static score. The chemistry prediction is that randomized-serialization training shrinks both the notation cost and the leakage inflation. A null result would show that the analogy breaks for stateful execution, which bounds how far molecular-augmentation lessons transfer.