When a code model "thinks step by step" about what a Python function returns, we usually cannot tell whether the steps produced the answer or were written afterwards to justify an answer the model already had. We may often be in the second case. Popular Python benchmarks are heavily leaked into training data, strong models are known to skip their own reasoning, and the same models look much weaker in Rust or Racket. This proposal argues that code execution is the one setting where the question can be settled cleanly. Every intermediate step of a program has a ground-truth value. We can therefore edit the model's stated intermediate value and check whether its final answer follows the edit. If it does, the reasoning did the work. If the model returns its original answer anyway, the reasoning was decoration, a pattern called post-hoc rationalization.
The core claim is that much of the gap between Python and other languages is a gap in whether the reasoning trace is doing the work, not only in accuracy. In familiar settings, where Python and leaked benchmark programs predominate, models answer from memory and the trace is decorative. In unfamiliar settings, meaning low-resource languages and freshly composed programs, the trace becomes load-bearing, and failures show up as genuine errors in chaining steps together. We test this by crossing three factors: language, familiarity, and composition depth. The crossing includes a controlled experiment that deliberately leaks answers in Python only. The result would separate three explanations for cross-language gaps that the literature currently conflates: memorization, tokenization, and failure to compose steps.
What exists and what is missing
Three lines of work motivate the proposal without answering it.
Multilingual execution benchmarks measure accuracy only. CRUXEVAL-X translates the 800 CRUXEval functions into 19 languages. It reports that execution reasoning transfers surprisingly well and that program length and variable count matter more than type system. It never inspects reasoning traces and never controls for contamination. Its parent, CRUXEval, found that CoT helps GPT-4 but hurts smaller models, which is an unexplained asymmetry that trace faithfulness could explain.
Contamination work is confined to Python. Quantifying Contamination finds verbatim MBPP solutions in StarCoderData and explicitly leaves cross-lingual contamination unsearched. DyCodeEval and LiveCodeBench build contamination-resistant evaluations in Python. None of the three asks what memorization does to the reasoning trace.
Faithfulness work supplies the measurement ideas but uses natural-language or math tasks. On the Hardness of Faithful Chain-of-Thought Reasoning measures faithfulness by truncating the reasoning and checking whether the answer is already fixed, a method called early answering. It reports that more capable models can be less faithful. Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation defines three kinds of anchoring and a skeleton-guided mitigation called SSR. Can Aha Moments Be Fake? separates true steps from decorative ones. Decodable but Not Faithful is the nearest work in code: it shows internal states can match a verifier 98.6% of the time while the prose is unfaithful. That work studies single algorithmic claims, not multi-step execution, and not language or contamination. We reuse early answering as a baseline metric. What we add is an intervention metric grounded in true program state, which none of these papers has available.
The composition axis borrows from Irreducible Supervision Enables Compositional Generalization in Post-Training. That paper shows that depth-controlled programs built from typed primitives expose whether a model composes steps or uses shortcuts. Its result constrains our design. Programs must be irreducible, meaning no step can be skipped by a pattern shortcut; otherwise a correct final answer tells us nothing about composition.
Research Program
Program generator. We build a generator of execution tasks from about 40 typed primitives drawn from the standard-library operations CRUXEval uses: string slicing, list filtering, dictionary updates, arithmetic folds, sorting with keys, and similar. Each primitive gets a hand-written implementation in Python, Java, C++, Rust, and Racket. A composed program is then semantically identical across languages by construction. This avoids the heuristic type inference and test translation that the file flags as a source of artifacts in transpiled benchmarks.
Programs have composition depth from one to five. We filter them for irreducibility in the sense of the irreducible-supervision paper: we reject any program whose output equals the output of a shallower program obtained by deleting a step. Every program comes with its execution trace, the value after each primitive.
Familiarity axis. Familiarity has three levels. The first is the original CRUXEval programs and their CRUXEVAL-X translations, which are plausibly contaminated. For StarCoderBase-15.5B, whose training data is public, we measure this directly with the surface and AST similarity search of Quantifying Contamination, extended to non-Python splits. The second level is freshly composed programs that match CRUXEval's length and variable-count distribution. The third is a controlled leak, described below.
Structured trace prompt. Models are asked to state the value after each step in a fixed slot format ("after step k: v"), in the spirit of SSR's skeleton. This lets us score each step against the true trace and edit individual slots.
Metrics. Beyond final accuracy we compute three quantities. Step accuracy is the fraction of stated intermediate values matching the true trace. The right-for-wrong-reasons rate is the fraction of correct final answers that follow at least one wrong intermediate value. The counterfactual follow rate is defined as follows. We replace the model's stated value at step $k$ with a different value that is type-valid, re-execute the remaining program from that value to get the implied output $y'_k$, let the model continue generating from the edited prefix, and record whether it outputs $y'_k$:
$$ F = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\left[\hat{y}_i(\text{edit at } k_i) = y'_{k_i}\right] $$
Here $N$ is the number of edited instances, $k_i$ is the edited step, and $\hat{y}_i$ is the model's final answer after the edit. $F$ near 1 means the trace drives the answer. $F$ near the rate at which the model returns its original answer means rationalization. Because $y'_k$ comes from actual execution, $F$ needs no judgment about what a faithful answer "should" be.
Models. We use open-weight models so that the prefixes can be edited: StarCoderBase-15.5B (public training data), DeepSeek-Coder-33B, Qwen2.5-Coder-7B (which DyCodeEval flags as scoring much higher on static seeds than on variants), Qwen3-8B, and Llama-3.2-1B. Closed models are evaluated for accuracy and early answering only, since their prefixes cannot be edited.
Predictions. We expect $F$ to be lowest for Python on original CRUXEval programs and highest on fresh programs in Racket and Rust. On fresh programs, accuracy should fall with depth mainly through step errors whose effects carry forward, which is the signature of genuine composition failure. Within a language, the drop in $F$ between seed and fresh programs estimates how much of the apparent "reasoning" is memory.
Baselines
The naive reference is direct answering without reasoning, scored by accuracy on the same programs, languages, and sampling budget. If direct accuracy on seed Python programs matches CoT accuracy while $F$ is low, that is itself evidence the trace is decorative. For context, CRUXEval reports GPT-4 with CoT at 81.9% on output prediction and Code Llama 34B at 46%. CRUXEVAL-X reports GPT-4o at about 70% Pass@1 across its languages. We rerun these accuracy measurements on our models rather than comparing across papers.
The strongest existing faithfulness measurement is early answering: the area over the curve of final-answer probability as the reasoning is truncated, from On the Hardness of Faithful Chain-of-Thought Reasoning. It is the natural competitor because it is cheap and needs no ground truth. We compute it on exactly the same generations used for $F$, so the comparison is fair. We found no reported early-answering result for code execution on Lacuna, so there is no published number to beat. Our intervention metric is worth having if it separates conditions that early answering does not. The key case is a model that commits early but copies the intermediate values it states correctly. On such instances early answering calls the trace unfaithful, while $F$ checks whether the stated values actually steer the answer. The metric is also worth having if it predicts held-out accuracy under leakage better.
The strongest existing mitigation is SSR from Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation, which reports up to +9.1 points on ArenaHard. We apply its plan-then-infer skeleton with the same token budget. Success for SSR would mean raising $F$ on familiar Python programs without lowering accuracy on fresh ones.
We run two ablations. The first isolates the trace-grounded edit by comparing $F$ against a random-token edit, which checks whether models merely react to any perturbation. The second isolates contamination by holding programs fixed and varying only whether they were leaked.
Why not just…?
Why not just run CRUXEVAL-X or LiveCodeBench's execution scenario on fresh programs and compare accuracies? That would show that accuracy drops on unseen programs, which DyCodeEval already established for generation in Python. It cannot say whether seed-program answers came from reasoning or recall, or whether the drop in Rust is a composition failure rather than lost memorization. The step-accuracy and follow-rate measurements are exactly what an accuracy table lacks.
Why not just use early answering, which needs no edits? This is a real threat. If early-answering area ranks all language × familiarity cells the same way $F$ does, the cheaper metric suffices, and we will report that as a finding. We expect divergence for two reasons. First, early answering conflates "knew the answer early" with "ignored the trace," and the two differ when the model's early guess comes from partially computing steps internally. Decodable but Not Faithful shows that internal computation and the written account can come apart. Second, truncating a trace in a low-resource language may break the model's fluency rather than measure dependence. We compute rank agreement between the two metrics per cell as an explicit check.
Why not just blame tokenization? CRUXEval documents failures from tracking individual characters under subword tokens, so this is plausible for accuracy. It does not predict a change in $F$, because tokenization should hurt step accuracy equally on seed and fresh programs. We also include character-heavy and character-free primitive subsets and regress outcomes on tokens per program. Any residual language effect after these controls is not tokenization.
Why not just measure n-gram contamination and correlate it with accuracy? Quantifying Contamination already shows accuracy rising with similarity, from 72% on the most similar tenth of MBPP to 22% on the least similar tenth. That is correlational and silent about the trace. Our leak experiment makes the manipulation causal and measures its effect on the trace directly.
Decisive Experiments
Controlled Python-only leak. We fine-tune Llama-3.2-1B and Qwen2.5-Coder-7B on input-output answers for a random half of the fresh programs, in Python only and with no reasoning traces. We then evaluate the leaked and held-out halves in all five languages. This mirrors DyCodeEval's leak setup, where static scores rose from about 20% to about 80% while dynamic variants stayed near 13%, but here we read out the reasoning trace. The hypothesis is confirmed if leaked Python items show high accuracy with a sharp fall in $F$ and a higher right-for-wrong-reasons rate, while held-out items stay unchanged. The pattern in other languages decides whether memorization crosses languages, a question the contamination literature explicitly left open. Leaked Rust items gaining accuracy with low $F$ would mean the model recalls answers regardless of syntax. Leaked Rust items staying at baseline with high $F$ would mean memorization is tied to the surface language.
Composition depth on fresh programs. Plotting step accuracy, carried-forward error, and $F$ against depth for each language tests whether low-resource languages fail by composing steps badly (high $F$, errors compounding with depth) or by guessing (low $F$ at all depths).
Natural contamination for StarCoderBase-15.5B. We correlate per-program similarity to StarCoderData, in every language, with $F$. If similar programs show lower $F$, the leak-experiment result generalizes to real pretraining.
Mitigation test. Finally, we test whether SSR raises $F$ in the familiar cells. If it does not, rationalization driven by memorization is a regime that skeleton prompting cannot reach, which would point future mitigation work toward training-time fixes.
References
- CRUXEVAL-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
- CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation
- Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought
- Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers
- Irreducible Supervision Enables Compositional Generalization in Post-Training