Code models write programs as flat token streams, and they routinely lose track of what the program they are writing actually does. They call methods that do not exist on the object at hand, reuse names that are out of scope, and ignore a variable computed three lines earlier. One obvious fix is to run the partial program and show the model the result as an execution trace, meaning the record of variable values and types as the code runs. A careful study found this surprisingly unhelpful. When traces are pasted into the training data as text, models become more confident but not more correct, and they are often worse than models trained without traces (Do Code Semantics Help?). The one method that does turn execution into large gains, EG-CFG (Execution Guided Line-by-Line Code Generation), pays for them with test suites and with repeated lookahead rollouts and sandbox runs at every line.
We propose a third route. Take the live runtime state at the current line, together with the program's data-flow structure, and turn it into a soft bias on the next-token scores. The bias favors identifiers that are in scope and defined but not yet used. It also favors attribute and method names that are valid for the receiver's actual runtime type, and it penalizes names that do not exist. The argument is that the output vocabulary is the one space where program state and code tokens are already aligned. Injecting state there avoids the cross-modal representation gap that makes prompt-level traces fail. It needs one incremental execution of the prefix per line, with no rollouts and no expected outputs.
Core Claim
Our hypothesis is that execution traces fail as prompt text because of how they are delivered, not because they carry too little information. Work on multimodal translation found that a secondary modality supplied as global context gets ignored by the autoregressive decoder (A Visual Attention Grounding Neural Model for Multimodal Machine Translation). Work on multimodal recommendation argues that forcing heterogeneous signals through one sequential channel creates integration bias (FindRec). Both suggest the trace is being neglected or misread, not that it is worthless. If that is right, then delivering the same trace information at the token positions where it matters should convert it into correctness. The gains should concentrate on name, attribute and type errors. They should also disappear when the state comes from a different input's execution.
This borrows the topic-bias formulation from dialogue and translation, where a bias vector over the vocabulary is added to the model's scores so the output leans toward a theme (Xing et al., Wei et al.). Here the "topic" is not a semantic theme but the set of program entities that are live at this instant. We also borrow the gated "probability surgery" of RESEAL (Relation-Constrained Decoding for Text Generation), which boosts tokens that satisfy parser-identified relations without zeroing out other tokens. RESEAL checks static natural-language dependency triplets. We check dynamic program state and data-flow edges, and we do it without per-beam parser lookahead.
Research Program
At each line boundary, the partially written function is executed once on an available input, resuming from a snapshot of the interpreter's state at the previous line. The same prefix is parsed into a lightweight data-flow graph, of the kind GraphCodeBERT builds into attention (GraphCodeBERT), but computed only locally. The input need not come with an expected output. It can be an example from the docstring or a model-synthesized argument, which removes EG-CFG's need for a correctness oracle. From the snapshot and the graph we build a bias vector $\mathbf{b}_t$ over the vocabulary. Decoding then uses
$$P(w_t \mid w_{<t}) = \mathrm{softmax}\big(\mathbf{z}_t + g_t\,\mathbf{b}_t\big),$$
where $\mathbf{z}_t$ are the model's scores and $g_t \in [0,1]$ is a gate. The gate is nonzero only when most of the model's own probability mass falls on identifier-like tokens. This keeps the bias away from syntax, following the observation that soft biases should tip choices among plausible alternatives rather than override grammar.
Entries of $\mathbf{b}_t$ come from three sources. There is a positive weight for names that are defined but not yet consumed along the data-flow graph, and a positive weight for attributes valid on the receiver's runtime type, such as a DataFrame versus a list. There is a negative weight for identifiers that exist in neither the scope nor the runtime namespace. Multi-token identifiers are handled with a prefix trie, so the bias continues across subwords once a name is begun.
We build two versions. The first is rule-based, with three scalar weights tuned on a validation split. The second is learned. It follows the ITFormer recipe of training a tiny projection (under 1% of parameters) that aligns a temporal signal with a frozen language model (ITFormer). Our projection maps an embedding of the per-line state sequence to $\mathbf{b}_t$, trained on the verified executable programs of PyX (SemCoder) to raise the likelihood of the ground-truth next tokens. The learned version tests whether the trace carries signal beyond hand-written scope and type rules, for example which live value is numerically relevant.
The analysis half asks where the gains come from. We classify failures of every run by Python exception class and by whether a generated name or attribute was ever live. We measure calibration, meaning whether token confidence tracks correctness, to test directly whether logit delivery breaks the confidence-without-correctness pattern of prompt traces. We also report wall-clock time and sandbox calls per problem.
Baselines
The naive reference point is the same open ~7B code model, DeepSeekCoder-6.7B as used by SemCoder, decoded greedily and with temperature sampling and given no execution information. For published anchors, SemCoder-S reports 79.3% on HumanEval (SemCoder).
The strongest existing method is EG-CFG (Execution Guided Line-by-Line Code Generation). It samples lookahead completions at each line, patches them into executable form, runs them against tests, and interpolates guided and unguided scores. It reports 96.6% on MBPP, 89.02% on HumanEval-ET and 69.9% on DS-1000, all with DeepSeek-V3. Those numbers are not comparable to a 7B model, so we re-run EG-CFG on our backbone under the same problems. We run it twice: once with the provided tests, and once with only the same inputs our method gets, where "test" pass/fail is replaced by the absence of exceptions. We also run a version with a matched wall-clock budget, meaning EG-CFG gets as many rollouts as fit in our method's time. Beating it means matching its pass@1 within noise at a fraction of the sandbox calls, or exceeding it in the no-expected-output setting. We expect EG-CFG to keep an edge when real tests are available. Counting that as a loss on accuracy is the honest framing.
The prompt-trace baseline gives the model the identical state information as concise text. "Concise" is the only format that was non-detrimental in 11 of 14 test-time cases (Do Code Semantics Help?). We use it both at inference and in a fine-tuned variant on the same PyX data. The same study reports repair gains of only 0.3–1.4% from SemCoder traces, which frames what a meaningful improvement over prompt delivery looks like.
The key ablations isolate each ingredient. A static-only bias uses scope and type inference with no execution, which isolates the temporal component. A shuffled-state bias is built from executing a different problem's input, which checks that the gain comes from correct state rather than a generic "prefer known names" prior. A hidden-state injection variant shifts decoder states with the same state embedding, following Wang et al., which isolates the choice of the vocabulary as the injection point. All variants share the backbone, the problems, and a fixed budget of validation runs for tuning the weights.
Why not just…?
Why not just put the trace in the prompt, since long contexts are cheap? Direct evidence says this mostly raises confidence without raising correctness (Do Code Semantics Help?). SWE-bench also documents accuracy falling as prompts fill with extra code context (SWE-bench). Still, that study tested fine-tuning and test-time scaling, not per-line fresh state at inference. Our prompt-trace baseline gives exactly the same information per line. If it matches logit delivery, our central hypothesis is wrong, and we will report it that way.
Why not just run EG-CFG with model-generated tests? This is the most serious threat. We cannot rule it out from the literature, because the file gives no result for EG-CFG with synthesized tests. Generated tests carry wrong expected outputs, so guidance can push toward wrong behavior. Our method uses only inputs, never expected outputs, so a bad synthesized test cannot mislead it. EG-CFG also pays for rollouts at every line. The budget-matched and synthesized-input EG-CFG runs settle this directly.
Why not just constrain decoding with static scope and type analysis? For typed, attribute-heavy code this could capture most of the benefit. That is why the static-only ablation exists. We expect runtime state to matter most in dynamically typed library code, where static inference cannot tell a DataFrame from a Series or a list from a dict, which is the regime of DS-1000. If static-only matches our method on DS-1000, the execution component is unnecessary and the contribution shrinks to a cheaper constrained decoder.
Why not just generate, execute and repair afterward? Post-hoc loops let errors compound across a whole block before any feedback arrives, as EG-CFG argues. In agentic settings they are expensive: SynFix reports $3.34 and more than 323,000 tokens per issue for CodeR, versus $0.56 for structured dependency guidance. We include a sample-execute-rerank baseline at matched sample count. Our bias is compatible with it, so we also report the combination.
Proposed Experiments
We evaluate generation on MBPP, HumanEval, their extended-test variants MBPP-ET and HumanEval-ET, and DS-1000, reporting pass@1 and pass@10. We report latency and sandbox calls per problem. We also report the rate of NameError, AttributeError and TypeError failures, and calibration of token confidence against correctness.
For a repair setting we use HumanEval-R and MBPP-R from Do Code Semantics Help?. There the executed buggy program supplies the state, which lets us compare directly to that paper's trace-fine-tuning result. TraceFixer's CodeNet setup (63% top-1 in TraceFixer) serves as a secondary check that decoding-time bias can substitute for a supplied desired state.
Three outcomes decide the project. First, logit-level delivery must beat prompt-level delivery of identical state on pass@1. If it does not, the modality-neglect explanation fails. Second, the gain must shrink substantially under shuffled state, which shows the model uses correct runtime information and not a name prior. Third, the execution-based bias must beat the static-only bias on DS-1000. If it does not, the temporal signal is not earning its cost. If all three hold and the method approaches re-run EG-CFG at a small fraction of its sandbox calls, we will have shown that execution information helps code models when delivered at the vocabulary, even though it fails as prompt text.
References
- Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
- Execution Guided Line-by-Line Code Generation
- A Visual Attention Grounding Neural Model for Multimodal Machine Translation
- FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation
- Topic Aware Neural Response Generation
- Translating with Bilingual Topic Knowledge for Neural Machine Translation
- Relation-Constrained Decoding for Text Generation
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning
- Friendly Topic Assistant for Transformer Based Abstractive Summarization
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- SynFix: Dependency-Aware Program Repair via RelationGraph Analysis
- TraceFixer: Execution Trace-Driven Program Repair