Language models that solve problems by writing and running Python increasingly work in a persistent interpreter, where each new cell inherits everything earlier cells left behind. That shared state creates a classic software hazard called shadowing. Shadowing happens when a name the model or its environment defined earlier hides a standard one. One example is a variable called sum or max. Another is a file called random.py sitting in the working directory, so that import random no longer loads Python's own module. The resulting failure shows up far from its cause. The error message points at the line just written, while the real cause sits several cells back or on disk. We propose to use shadowing as a controlled, mechanically checkable probe of a specific reasoning weakness: can a model trace a failure back to a cause that is not in the code it just wrote?
Our core claim has two parts. First, there is a measurable gap in repair success between failures whose cause is local and failures with the same symptom whose cause is non-local, meaning it lies in inherited state. Second, the gap can be split into an information-access part and a reasoning part. The information-access part should close when the interpreter reports where the offending name was bound. The reasoning part should remain even then. We also test a quieter variant that matters for correctness. Here the shadow is type-compatible, for example a local math.py defining pi = 3.14, so no error is raised at all. The question is whether code-based self-verification catches the wrong answer or simply inherits it.
Why this intersection, and what is already known
On the software side, shadowing is common and well documented. ModuleGuard uses syntax-tree parsing and simulated installation to show that over 8% of PyPI packages conflict with the Python standard library and 21.45% with other third-party libraries. Its stated limitation is that it cannot tell whether a shadowed path is ever actually executed. Peng et al. show that local project files silently shadow external modules in ways dependency checkers miss. Tang et al. document the same pattern in Android crash reporting: a single global slot means the last registered library silently overwrites the earlier ones. In grammar engineering, Parmentier et al. find that shared global identifiers cause distinct fragments to be wrongly merged. These papers motivate the hazard. They study human-written software and say nothing about whether a model can diagnose the hazard once it occurs.
On the language-model side, interpreter use clearly helps. PAL offloads computation to Python and holds 61.5% on GSM-HARD where chain-of-thought falls to 20.1%. Zhou et al. add code-based self-verification and reach 84.32% on MATH. They report that a "True" verdict from the model's own checking code is correct 95.88% of the time. CIBench evaluates 10 to 15 step notebook sessions and attributes 40.4% of GPT-4's failures to reasoning errors and only 4.3% to API hallucinations. CIBench treats the runtime as a black box, so it cannot tell us whether any of those "reasoning errors" are really failures to attribute errors to inherited state.
Two further papers constrain how we expect models to behave. Al-Khalili et al. show that final-answer accuracy hides unsound program logic. This warns us that answer accuracy alone will not reveal silent shadows. Zhe Yang shows models failing easy variants of tasks they can do when small irrelevant changes are introduced. This suggests that an innocuous file in the working directory could derail an otherwise competent model.
No paper in the file connects these two lines. The closest systems work, IsolateGPT, isolates whole agent applications from each other for security. It does not study name binding inside a single interpreter, and it does not ask whether the model can reason about that binding.
Research Program
The work has three pieces, built on one instrumented Python executor. The executor records, for every name lookup and every import, which cell or file supplied the binding. It does this through import hooks and tracing of global-variable writes. Because it observes execution directly, it resolves ModuleGuard's reachability question for our setting: we count a shadow only when the shadowed binding is actually used.
The first piece is a prevalence audit. We rerun PAL-style prompting on GSM8K, GSM-HARD and MATH500, code-based self-verification on MATH, and full CIBench sessions. Models are drawn from those used in the file: Llama-3-70B-Instruct, the MathCoder2 7–8B models (resource), CodeLlama-34B, Qwen and GPT-4o-mini. We count how often models shadow names in each of three ways: by assigning to built-in names, by writing files whose names match standard modules, or by reusing names between the generation and verification blocks. We then measure how often each kind of shadow is actually reached at runtime and what fraction of failed sessions it accounts for. This sets the practical stakes. If natural shadowing is rare, we will say so, and the remaining pieces stand as a diagnostic probe rather than a fix for a common failure.
The second piece is a matched-pair injection benchmark, which carries the main scientific contribution. We start from CIBench sessions and PAL programs that the model completes successfully. For each, we construct a pair of perturbations that produce the same exception type and the same failing line, but differ in where the cause lives. In the local version, the bug is in the current cell. In the non-local version, the cause is a binding introduced three or more cells earlier, or a file placed in the working directory. An example of the non-local case is a statistics.py stub that makes statistics.median raise an AttributeError. We also build silent variants, where the shadow returns plausible wrong values and raises nothing. We measure three things: whether the model repairs the failure within a fixed turn budget, whether its stated diagnosis names the true cause (checked mechanically against the executor's binding log), and how often a silent shadow produces a wrong final answer.
The third piece is an intervention that separates access from reasoning. Here the executor appends a one-line note to every error report, stating where the failing name was bound, for example "median resolved from ./statistics.py, not the standard library". If this note closes the gap between local and non-local repair, the limitation is mainly that the model lacks visibility into interpreter state, and the fix is engineering. If a large gap remains even with the binding location in plain view, we have evidence of a genuine attribution weakness. That would be consistent with the self-planning deficits Cai et al. observe, where models execute well but plan poorly.
Baselines
The naive reference point is the model's own unmodified repair loop: it reads the standard error message and tries again. This is the setting CIBench evaluates, and every other condition is measured against it. We give it the same turn budget and sampling settings as every other condition.
The strongest existing method for catching wrong answers is code-based self-verification with verification-weighted voting from Zhou et al.. It reports 84.32% accuracy on MATH and 95.88% precision of "True" verdicts. It is the right competitor for the silent-shadow condition because it is the published mechanism most likely to catch a wrong value without any knowledge of namespaces. We run it with the same number of sampled paths across our models. We say it is beaten if its verdict precision on silent-shadow items falls clearly below its clean-item precision while the provenance condition stays high. For the multi-turn setting, the reference is plain CIBench behaviour. The file reports no CIBench numbers under injected shadowing, so we state: no reported result found on Lacuna for shadowing-specific repair rates. Our own clean runs supply the comparison point.
The engineering baseline is a static guard run before every cell executes. It combines the linter warning for redefined built-ins with a working-directory scan that resolves each import's origin file. This guard prevents shadows rather than diagnosing them, and it is the cheapest serious competitor. The key ablation is the provenance note itself, compared against an equal-length note that gives no binding information. This controls for the model simply paying more attention to longer error messages.
Why not just…?
Why not just tell the model not to reuse built-in names? This addresses model-introduced shadows but cannot affect shadows the environment introduces, such as files left by data downloads or earlier tools. Prompt-level self-guidance has also underperformed in related settings: Cai et al. find that self-guided planning degrades code generation relative to direct generation. We include this instruction as a condition in the audit rather than assuming it fails.
Why not just run a linter and an import-origin check before each cell? This is a real threat for prevention, and we expect it to eliminate most natural shadows cheaply. If the audit shows it does, we will recommend it. But it answers a different question. Our scientific claim concerns whether models can attribute failures to non-local causes. The benchmark holds the symptom fixed while moving the cause, which is exactly what a prevention tool sidesteps. A guard also gives no help with silent shadows that arise through state rather than names, such as a mutated module attribute.
Why not just restart the kernel every step? CIBench tasks require persistent state across steps, so restarting forces the model to regenerate earlier work. A restart also leaves files on disk untouched, so file-based shadowing survives it. We report a restart condition so that its repair-rate and token-cost tradeoff is visible rather than argued.
Why not just avoid code and use chain-of-thought? PAL reports chain-of-thought falling to 20.1% on GSM-HARD against 61.5% for program-aided reasoning, and Kabra et al. report roughly halved calibration error with code. Abandoning the interpreter gives up far more accuracy than shadowing plausibly costs.
Decisive Checks
Three results would settle the proposal. The first is a significant repair-rate gap between the local and non-local versions of each matched pair under the naive loop, across model sizes. Without that gap there is no attribution limitation to study. The second is the size of the gap that remains after the provenance note is added, which tells us whether the limitation is about access or about reasoning. The third is the silent-wrong-answer rate under self-verification when verification shares a namespace with generation, compared with running verification in a fresh namespace that receives only the final answer. If precision on verified answers drops only in the shared condition, then the strong published verification precision rests partly on circular checking. That would be a concrete, actionable finding for anyone building interpreter-based reasoning systems.
References
- ModuleGuard: Understanding and Detecting Module Conflicts in Python Ecosystem
- Less is More? An Empirical Study on Configuration Issues in Python PyPI Ecosystem
- A Systematical Study on Application Performance Management Libraries for Apps
- XMG - An Expressive Formalism for Describing Tree-Based Grammars
- PAL: Program-aided Language Models
- Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
- CIBench: Evaluating Your LLMs with a Code Interpreter Plugin
- Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems
- Beyond Code: Evaluate Thought Steps for Complex Code Generation
- Program-Aided Reasoners (Better) Know What They Know