Coding agents that fix their own programs by running them and reading the error messages, a practice called iterative code refinement, carry a hidden assumption. When a call fails, they treat the failure as their own logic mistake. Very often the real cause is that the model's memory of the library is out of date. The function was renamed, moved, or removed in the version actually installed. Models trained on a decade of mixed GitHub code cannot tell which version a remembered call belongs to, and simply telling them the version does not help. We propose to reverse where the loop places its trust. The model's recollection of any external API should count as unknown until checked. The installed environment, whose modules and function signatures can be listed completely by looking inside the running interpreter, should count as the authority whose silence means absence.
In short: before and during repair, every external call the program depends on is checked deterministically against the live library and recorded as confirmed, refuted, or unchecked. A refuted call switches the agent from debugging into migration. When the live library offers no replacement, the agent stops and reports that the task cannot be done in this version rather than patching forever. We expect this to lower the rate at which anachronistic calls persist and to reduce the defensive "slop" that accumulates when agents repair code against a library that has moved on. We treat the rival possibility, that an agent handed ordinary introspection tools or a type checker does just as well, as the main hypothesis to beat.
Core Claim
Two literatures have each examined half of the problem. LibEvoBench measures anachronistic errors across 29 pinned versions of PyTorch, NumPy and SciPy. These are calls that are syntactically valid and were correct in an older release but are wrong in the target release. The paper reports that adding a version tag to the prompt gives "zero measurable improvement" and that models from 35B to 397B parameters follow the same decay curve. It evaluates one-shot generation only, with no execution loop. Refinement systems such as OpenCodeInterpreter and ARCS feed tracebacks back to the model, but they assume the environment matches the model's beliefs. So nobody has measured whether execution feedback actually repairs anachronisms or only circles around them.
The conceptual lever is old. Levesque's formal treatment of incomplete knowledge bases separates "φ is false" from "φ is not known" using a knowledge operator. Gelfond's Strong Introspection shows that negation-as-failure, which treats "could not prove it" as "it is false," collapses that distinction. Our observation is that code environments contain both kinds of world. The model's parametric API memory is an open world: absence of doubt is not evidence. A pinned, installed package is a genuinely closed world, because Python can enumerate its namespace and bind signatures. So the closed-world assumption is not an error to remove. It should be applied to the source that can support it.
For each external symbol $a$ referenced by the program, with installed environment $E$, the loop keeps a three-valued status:
$$ s(a) = \begin{cases} \text{confirmed} & a \in \mathrm{ns}(E) \text{ and the call binds to its signature} \\ \text{refuted} & a \notin \mathrm{ns}(E) \text{ or binding fails} \\ \text{unchecked} & \text{not yet resolved, e.g. dynamically constructed} \end{cases} $$
Here $\mathrm{ns}(E)$ is the set of importable names, obtained by introspection. Only refuted symbols trigger migration. Only an exhausted search of $\mathrm{ns}(E)$ for a functional replacement licenses abstention.
The closest existing work is FUn-FuSIC. It repairs SPARQL queries iteratively, using "strong verifiers" that reject relations absent from a static knowledge-graph schema, and declares "No Knowledge" when an answer is missing. It reports 76.6 F1 on WebQSP→GrailQAbility against 58.4 for the previous best system. We borrow its verifier-then-abstain structure openly. We differ in three ways. The schema is not given but recovered by runtime introspection. The schema changes across versions, and that change is the object of study. And a changed signature must be propagated through downstream imperative code, whereas a single query can simply be rejected.
Research Program
The work has three parts.
The benchmark. We will build VerDrift-Repair from LibEvoBench's pinned versions and from the 145 verified deprecated-to-current mappings in EDAPIBench. Each instance is a short program with unit tests that passes under an older version $v_{\text{old}}$. The agent must make it pass under $v_{\text{new}}$ inside a Docker sandbox with $v_{\text{new}}$ installed.
Instances fall into three strata. In the rename stratum, a drop-in replacement exists, as in torch.svd → torch.linalg.svd. In the contract-change stratum, the replacement has different arguments, return shapes or semantics, so downstream code must change too. In the no-replacement stratum, the capability was removed. The correct output for this stratum is a structured abstention naming the missing symbol and the version. We will confirm it by exhaustively checking that no public symbol in $v_{\text{new}}$ passes the tests under a reference migration written by annotators.
A matched control set of version-neutral bugs, ordinary logic errors in stable APIs, checks that the method does not hurt normal debugging.
The loop. A static pass extracts every attribute chain rooted in an imported module. The ledger resolves each one against the live interpreter using import, attribute lookup and signature binding. This runs before the first execution and again after every patch. Refuted entries are shown to the model as explicit contradictions ("your belief numpy.float is refuted in numpy==X; candidate symbols in this namespace: …"), with candidates ranked by name and docstring similarity. This prevents the model's stale belief from being silently reinforced.
When a replacement changes a contract, a StateMem-style dependency pass, following Can Agent Memory Systems Track Evolving State?, marks every downstream variable and call derived from the changed value as stale and requires it to be rechecked. That paper maintains dependency links between remembered facts and flags derived facts when a source changes, in dialogue memory. We reuse the mechanism on program data flow. Abstention is allowed only when the ledger holds a refuted symbol, the candidate search is exhausted, and tests still fail.
The analysis. We will measure the anachronism persistence rate: the fraction of trajectories that re-emit a refuted symbol, or another stale alias of it, after a traceback has already exposed it. This number directly tests the claim that raw feedback leaves models trapped.
Baselines
The do-nothing reference is one-shot generation with the target version pinned in the prompt, which LibEvoBench already predicts will help nothing. It sets the floor.
The raw-traceback loop is an ARCS/OpenCodeInterpreter-style synthesize–execute–repair cycle with the same iteration budget $B$, the same backbone and the same sandbox. It is the default practice, and the reviewer's claim that "the interpreter already says AttributeError" is its whole case. For scale, OpenCodeInterpreter-33B rises from 79.3% to 82.9% HumanEval pass@1 with execution feedback. Those tasks involve no version drift, and no reported result exists on Lacuna for any refinement loop on version-drifted code, so we will measure this baseline ourselves.
The strongest competitor is an introspection-tool agent. It is the same loop, but the agent may call dir, inspect.signature, help and pip show whenever it chooses, and it also receives type-checker diagnostics against the installed package after each patch. It has the same information our method has, minus the forced and deterministic ledger. It gets an identical tool-call and token budget. Its prompts are tuned on the same development split with the same number of trials.
A documentation-RAG agent retrieves target-version docs for symbols named in tracebacks. LibEvoBench reports that supplying the exact target documentation adds 10–20 points in one-shot generation.
We also run a Structured Verification variant using the five-step checklist from Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?, which lifted Gemma-3-4B abstention F1 on ToolBench API replacement from 20.90% to 68.57%. It is the strongest known prompt-only abstention method.
Ablations remove, one at a time, forced ledger resolution (turning it back into optional introspection), downstream stale propagation, and the abstention gate.
Our method beats the field only if it outperforms the introspection-tool agent on tests passed within budget $B$ in the contract-change stratum, and on abstention F1 in the no-replacement stratum. We also require that it not lose ground on the version-neutral control set. We have no reported number for any of these methods on this task and will not assume one.
Why not just…?
Why not just give the agent introspection tools and a type checker? This is the real threat. If it matches our method, the contribution shrinks to the benchmark and the persistence measurement, and we will say so. Our reason for expecting a gap is indirect but specific. The ICT paper documents "action bias": agents act before checking whether conditions hold, even when checking is possible. The state-tracking paper finds that models anchor on salient stale facts even when the correct fact is in context, and that extra reasoning sometimes makes this worse. Both effects predict that agents underuse optional checks exactly when their stale belief feels confident. The comparison between forced and optional ledger use isolates this one ingredient.
Why not just retrieve current documentation? LibEvoBench shows that documentation helps when the right page is supplied. But choosing the right page in the first place requires already knowing which remembered call is stale, which is the job the ledger does. Documentation retrieval also has no principled stopping point for the no-replacement stratum. OpenIKLR reports that retrieval by semantic similarity supplies topically related but logically irrelevant context. OpenIKLR is a solver-guided loop that fills in missing premises for logic problems, and its 78.54% share of failures in translating language to logic warns us to keep our verifier mechanical rather than model-translated. We run documentation retrieval as a full baseline and also as an addition to our method, to test whether the two complement each other.
Why not just edit the model's weights for deprecated APIs? AdaLoRA-L exceeds 98% effectiveness on single-line EDAPIBench edits. Its authors, however, leave multi-line refactoring and sequential editing across many versions unsolved, and every user's installed version is different. We include an AdaLoRA-L-edited backbone inside the raw-traceback loop on the rename stratum, where it should be strongest, and expect it to fail on contract changes and removals.
Proposed Experiments and Decisive Checks
We will run all methods on two open code models and one API model, at iteration budgets $B \in \{1, 3, 5, 10\}$.
For each method and stratum we report five quantities: the fraction of tests passing, the anachronism persistence rate, abstention precision, recall and F1 on the no-replacement stratum, iterations to success, and the change in structural erosion and verbosity measured as in SlopCodeBench. SlopCodeBench found erosion rising in 77% of agent trajectories under changing specifications. We test whether drift-induced repair produces the same effect and whether the ledger reduces it.
Three results would settle the question. First, if the raw-traceback loop shows persistence near zero, the premise of this proposal is false. Second, if forced and optional introspection perform the same, the key ingredient adds nothing and the contribution is the benchmark alone. Third, if abstention F1 improves without lowering pass rate on the rename and control strata, that shows the open-world/closed-world split buys caution without timidity.
A final check on LibEvoBench's API-calling task, with the ledger allowed only one resolution pass, will show whether the gains carry over to the benchmark's own SEUS metric. The leading reported score there is 86.0, from GPT-5.4.
References
- LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
- ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement
- The Interaction with Incomplete Knowledge Bases: A Formal Treatment
- Strong Introspection
- Iterative Repair with Weak Verifiers for Few-shot Transfer in KBQA with Unanswerability
- Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations
- Can Agent Memory Systems Track Evolving State?
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- Open-World LLM Logical Reasoning
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement