Language models are increasingly asked to judge code without running it. Typical questions are "does this analysis script actually show that region A has the highest sales?" or "will this assertion pass?" When a model answers wrong, two different things may have gone wrong. It may have mis-simulated the program, holding a wrong picture of what the code computes. Or it may have simulated correctly and then drawn the wrong conclusion from its own picture. Current benchmarks report a single accuracy number and cannot tell these apart. The two failures call for opposite fixes: better state tracking in one case, better calibration and decision-making in the other.
We propose to borrow a decision-theoretic accounting from visualization research and apply it to mental code execution. The Rational Agent Benchmark for Data Visualization compares human performance against an ideal agent and splits the gap into belief loss and optimization loss. Belief loss is the failure to extract the information a chart contains. Optimization loss is the failure to act well on what was extracted. Our version treats the model's own sampled execution traces as its "belief." We then use two kinds of ordering manipulation as causal levers. The first reorders a program's independent statements, which preserves its semantics. The second reorders the sequence in which a model judges items within one session. Our central prediction is that these two levers load on different components. Statement reordering should inflate simulation (belief) loss and leave decision loss flat. Sequential carry-over between judged items should inflate decision loss and leave simulation loss flat. If that dissociation holds, the decomposition is a real diagnostic and not a relabeling of accuracy.
Core Claim
A code-judging error can be attributed, item by item, to simulation or to decision. The attribution compares three scores on a probabilistic judgment task. The task asks the model to report a probability $p$ that a claim $c$ about a program's output holds, scored by the Brier rule $S$, where higher is better and the score uses negated squared error. The first score, $R_V$, is the score of an ideal agent that knows the true output; it is the ceiling. The second, $R_\emptyset$, is the score of an ideal agent that knows only the base rate of true claims. The third, $R_B$, is the score of an ideal agent that reasons from the model's own beliefs. We obtain those beliefs by sampling $k$ explicit execution traces from the model, in the step-by-step state-reporting style of Code Simulation as a Proxy for High-order Tasks in Large Language Models. We then set $p^* = \frac{1}{k}\sum_j \mathbb{1}[c(\hat{s}_j)]$, the fraction of simulated outputs $\hat{s}_j$ under which the claim holds. Let $B$ be the score of the model's actual stated $p$. Then
$$ \underbrace{R_V - B}_{\text{total loss}} = \underbrace{(R_V - R_B)}_{\text{simulation loss}} + \underbrace{(R_B - B)}_{\text{decision loss}}, \qquad \Delta = R_V - R_\emptyset . $$
Here $\Delta$ is the value of knowing the execution result, which normalizes both losses so they are comparable across tasks. The identity is trivially true. The scientific content lies in whether the components respond selectively to interventions designed to hit only one of them.
Relation to Prior Work
Several papers motivate the gap. Execution benchmarks show that models mis-simulate code. CRUXEval measures output and input prediction on 800 short Python programs. CodeSense measures statement-level and block-level tracing in real repositories. DSR-Bench measures data-structure mutation. A separate line shows that models judge code poorly even when they can write it: NoFunEval, CodeJudge-Eval, and CodeScope. Neither line can say whether judging failures inherit from simulation failures, which is the question this proposal targets.
Other papers supply the methods. The rational-agent paper supplies the decomposition. It was designed for humans reading charts and has never been applied to models or to code. Premise Order Matters in Reasoning with Large Language Models supplies the permutation methodology and the Kendall tau measure of how far an ordering departs from the original. It showed drops of more than 30% on logic and math word problems, but it never tested program statements. Sequence Effects in Crowdsourced Annotations supplies the autoregressive regression used to detect carry-over between consecutive judgments. It also supplies the per-subject randomization principle: one shared random order across all subjects turns quirks into systematic bias. That paper studied human annotators only.
The closest work on perturbation is Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates, together with studies of obfuscation. These change surface form, through renaming and branch swaps, and measure accuracy drops. We instead use dependency-preserving reorderings and ask which component of loss they move.
One paper constrains the design: PAL. Offloading execution to an interpreter removes simulation error, so in our setting the interpreter serves as the source of $R_V$ and not as a competitor.
Research Program
The task suite has three tiers sharing one format: a program plus a checkable claim about its output, half true and half false. False claims are generated by perturbing the true output.
The first tier uses CRUXEval programs. The second uses Python functions from CodeSense, filtered to those with deterministic, serializable outputs. The third uses table-analysis code, which connects the work to automated insight generation. It draws on the Pandas question-code pairs and Open-WikiTable tables from Semantically Aligned Question and Code Generation for Automated Insight Generation. The claims in this tier are insight statements of the form "the result shows X is largest" or "the trend is increasing." This tier is where insight quantification becomes concrete: we measure how much of a model's failure to verify an analytic insight is simulation and how much is judgment.
Every program is executed once to obtain ground truth. For each program we then build a statement dependency graph covering data dependencies, control dependencies, and side effects on shared objects. We enumerate alternative topological orders and re-execute each variant to confirm that its output is identical. This yields a family of semantics-preserving variants indexed by Kendall tau distance from the original. Many CRUXEval programs are short and admit only one valid order. We therefore restrict the reordering analysis to programs with at least two valid orders and report how many survive this filter.
For sequential effects, items are presented in multi-turn sessions of twenty judgments with prior answers kept in context. Each session uses its own random order, following the per-worker randomization of the crowdsourcing study. We fit
$$ Y_{t} = \beta_0 + \beta_1 Y_{t-1} + \beta_2\, \text{Gold}_t + \eta , $$
where $Y_t$ is the outcome on the $t$-th item of a session, $Y_{t-1}$ is the outcome on the previous item, and $\text{Gold}_t$ is the claim's truth value. We fit it three times, with $Y_t$ taken as the stated probability, as the simulation-loss component, and as the decision-loss component. Our prediction is a nonzero $\beta_1$ for decision loss and approximately zero for simulation loss. To make this test possible, traces for computing $R_B$ are always sampled in fresh, context-free calls. Simulation beliefs are therefore insulated from the session by construction, and any carry-over must enter through the decision.
One worry is that asking for traces changes the model's behavior. We measure this directly by comparing $B$ for judgments made with and without a preceding trace. If elicitation itself shifts accuracy, we report the decomposition for both regimes rather than claiming that one is canonical.
We will run open models such as the Code Llama family, which lets us control sampling, and at least one frontier API model.
Baselines
The naive reference point is the prior-only agent, $R_\emptyset$, together with a direct judgment that uses no trace and no chain of thought. Any method that does not beat $R_\emptyset$ adds no information beyond the base rate.
The strongest existing in-model method is chain-of-thought judging. On the closest standard task, the CRUXEval paper reports GPT-4 with chain of thought at 81.9% on output prediction and 75.0% on input prediction, against 46% and 50% for Code Llama 34B (CRUXEval). We run chain-of-thought judging with the same models, sampling budget $k$, and prompt-tuning budget as our trace-based estimate.
On the insight tier, the strongest existing judge is the embedding-based alignment classifier, reported at 85.1% accuracy against 83.5% for GPT-4 (Semantically Aligned Question and Code Generation for Automated Insight Generation). That classifier answers a different question, whether code matches a question, so we re-train it on our claim-verification labels with the same training split.
There is no reported result found on Lacuna for any model under semantics-preserving statement reordering of CRUXEval or CodeSense. We will establish those numbers ourselves.
The interpreter oracle in the style of PAL is an upper bound and supplies $R_V$; it is not a baseline to beat. The key ablation is trace injection. We replace the model's sampled traces with the true trace and re-ask for the judgment. This should drive simulation loss to zero and expose whether decision loss survives when the model is handed the correct state.
"Beating" the baselines here does not mean higher judgment accuracy, which is not the goal. Success has three parts. Under reordering, simulation loss should rise significantly while decision loss stays within a confidence interval of zero. Under sequential presentation, the reverse should hold. And the decomposition should predict the effect of trace injection better than raw accuracy does.
Why not just…?
Why not just run the code in an interpreter? For deployment that is usually the right answer, and we do not dispute it. The interpreter is exactly how we obtain $R_V$. The question here is diagnostic. Judges are used when execution is unavailable: a source table is missing, side effects are unsafe, or a judge is scoring rendered output without access to the source, as in VisJudge-Bench. DSR-Bench shows that near-perfect scores with an interpreter coexist with unmeasured internal failure. An interpreter tells you the answer, not why the model would have got it wrong.
Why not just correlate simulation accuracy with judging accuracy across items? This is the most serious threat. If raw CRUXEval-style output accuracy predicts judging errors as well as our decomposition does, the decomposition adds little. Correlation pools items and cannot attribute an individual error. Specifically, it cannot detect cases where the model simulated correctly and still judged wrong, which is precisely what the generative-paradox papers suggest is common. We test this head-to-head. Both the correlation model and the decomposition are used to predict the accuracy gain from trace injection, item by item. If the correlation model matches the decomposition, we will report that the simpler analysis suffices.
Why not just use existing counterfactual perturbations such as renaming and branch swapping? Those perturbations change the lexical surface the model may have memorized, so they mix memorization with simulation. Dependency-preserving reordering leaves every token and identifier intact and changes only sequence. That targets the left-to-right bias documented for premises. Running both perturbation types lets us check whether they load differently on the two loss components.
Why not just randomize item order and move on? The crowdsourcing study shows that randomization converts carry-over into noise rather than removing it, and that noise of this size, a 0.18 standardized-unit contrast shift, can reorder leaderboards. Measuring which component of loss carries the effect tells us whether a judge's sequence sensitivity is a decision-layer artifact that could be fixed by resetting context.
Decisive Checks
The proposal fails cleanly in any of three ways. Reordering might raise decision loss as much as simulation loss, which would mean the components are not separable by intervention. Trace injection might leave most of the total loss in place on CRUXEval and CodeSense, which would mean judging failures are mainly decision failures and the "mental execution gap" framing is misplaced for judges. Session carry-over might appear in fresh-context simulation loss, which would signal a leak in the measurement. The positive outcome is a double dissociation across the three tiers. On the insight tier, that outcome would show for the first time what fraction of a model's failure to verify an analytic insight comes from mis-executing the analysis, as opposed to misreading what that analysis showed.
References
- The Rational Agent Benchmark for Data Visualization
- Code Simulation as a Proxy for High-order Tasks in Large Language Models
- CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
- CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
- Can LLMs Reason Structurally? Benchmarking via the lens of Data Structures
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness
- CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
- CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
- Premise Order Matters in Reasoning with Large Language Models
- Sequence Effects in Crowdsourced Annotations
- Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates
- PAL: Program-aided Language Models
- Semantically Aligned Question and Code Generation for Automated Insight Generation
- VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations