Program Bindings as a Diagnostic for Memorized, Stale, and Misread Answers

When a language model gets a reasoning problem right, the score alone does not tell us whether it computed the answer, recalled the benchmark's answer key, or happened to agree with a fact it learned years ago. When it gets the problem wrong, the score does not tell us whether it misread unfamiliar wording, used an outdated fact, or made an arithmetic slip. Program-of-thought prompting gives us a way to separate these cases. In this style of prompting, the model writes a short Python program that a real interpreter then runs, instead of reasoning in prose. Such a program states each quantity it uses as a named variable with an explicit value, for example wage = 7.25. We can compare every one of these values against three references: the problem text, a timestamped record of real-world facts, and the original benchmark item. That comparison shows where each wrong number came from.

The proposal is to build and validate this kind of "binding audit." We will create controlled variants of standard math and financial benchmarks, and we will deliberately plant memorized items and outdated facts into an open model so that we know the true cause of every error. Then we test whether reading the program's variable assignments identifies the cause per item better than the standard aggregate signals: accuracy drops, post-cutoff decay, and membership-inference scores. Language drift plays a supporting role. We rewrite problems in archaic and in slang-heavy registers. This tests whether the program's structure stays fixed when surface wording changes, and it also acts as a natural form of rephrasing, which Test of Time showed can hide contamination.

Core Claim

Program-of-thought is usually studied as a way to raise accuracy. Two original papers established this. Program of Thoughts Prompting reported an average gain of about 12% over chain-of-thought, the step-by-step prose reasoning style. PAL reported 72.0% on GSM8K with Codex and only a small loss on large-number variants: 61.5% on GSM-HARD, while chain-of-thought fell to 20.1%.

We claim the program is also a measurement instrument. Each variable assignment is a small, checkable statement about where a number came from. That makes it possible to attribute errors to individual items in a way that output accuracy cannot.

Concretely, every literal value in the generated program falls into one of five classes:

  • Context-faithful: the value appears in, or is derivable from, the given problem.
  • Answer-key leak: the value matches the original benchmark item but not the perturbed one we presented, which signals memorization.
  • Stale-fact leak: the value matches an earlier real-world value of a fact that the context has updated.
  • Hallucinated: the value matches none of the references.
  • Computed: the value is produced inside the program.

The hypothesis is that the frequency of each class detects planted memorization and planted outdated knowledge more accurately, item by item, than the aggregate signals listed above.

We also make a second, falsifiable prediction. The research file notes that program-of-thought hardcodes values as if they never change. We therefore expect programs to reduce answer-key leaks but not stale-fact leaks. A program that writes down an outdated value runs it through exact arithmetic and returns a confident wrong answer. If this holds, the field's assumption that program-aided reasoning is uniformly more robust is wrong in exactly the setting where knowledge goes out of date.

Prior work and how this differs

Three papers motivate the gap.

Test of Time found that the post-cutoff accuracy decay used as a contamination signal appears on verbatim cloze questions (−3.83 points for GPT-4o-mini) but vanishes once questions are rephrased or variables in LiveCodeBench are renamed. Their influence-function attribution fell from a 77.5% to a 17.5% hit rate. Aggregate temporal signals are therefore fragile. That paper treats code only as a benchmark target, never as the model's reasoning medium, and its attribution is too expensive to run beyond about 40 documents.

LLM-Symbolic Integration for Robust Temporal Tabular Reasoning showed that generating SQL cuts GPT-4o's original-versus-counterfactual gap on TEMPTABQA-C from 17.21 to 2.96 points. This is the closest evidence that code resists memorized answers. However, it measures only accuracy gaps on static synthetic tables and never asks which values the queries contained.

SituatedQA documented the "update fallacy": even with updated passages in context, models were 15 points less accurate on updated facts than on stable ones. That is exactly the stale-fact leak we want to locate inside programs.

Two papers supply methods we reuse.

When Do Program-of-Thought Works for Reasoning? constrains the design. Small models write weaker programs, so audit coverage, meaning the share of programs that run, must be reported separately from audit accuracy.

Research Program

Benchmark variants. We start from GSM8K, GSM-HARD, FinQA, and the CounterFact split of TEMPTABQA-C. For each source item we build three variants:

  • Value counterfactual: every number is altered while the structure is kept, extending the GSM-HARD construction.
  • Fact update: the problem depends on a real-world quantity that has changed over time, such as a statutory wage, a tax rate, an exchange rate, or an infobox attribute. The context states the current value and the model is told to use it. We restrict these to facts with documented change histories so the stale value is known.
  • Register shift: the problem is rewritten either in Early Modern English spelling and syntax or in contemporary internet slang. The resulting text is checked by annotators for unchanged meaning.

The register variants draw on the drift characterization in Part-of-Speech Tagging for Historical English, where tagging accuracy falls from about 97% to 74.15% on Early Modern English, and in TimeLMs.

Planted ground truth. Black-box models cannot tell us why they erred, so we create the causes ourselves on OLMo-2-7B-Instruct. This is the model Test of Time used for attribution, which lets us cross-check against their influence method on a subset.

We fine-tune three copies:

  1. One on a random 20% of test items with their answers, to plant memorization.
  2. One on dated statements of old values of the update facts, to plant obsolescence.
  3. One on both.

A fourth copy is fine-tuned on matched unrelated text to control for the effect of fine-tuning itself. Each item then has a known label of "planted memorized," "planted stale," or "clean."

The audit. We run the generated program and extract each literal assignment from its syntax tree, which is the program's parsed structure. Each literal is matched against the context, the original item, and the dated fact store, with numeric tolerance. The item-level score is the fraction of bindings in each class. Detection is scored by area under the ROC curve (AUROC, the probability that a randomly chosen planted item is ranked above a randomly chosen clean one) against the planted labels.

Baselines

Naive reference. The do-nothing reference is the accuracy gap between the original and perturbed versions of an item, used as a per-item flag. This is the signal behind GSM-HARD and the CounterFact analysis. Its reported strength is the 17.21-point gap for GPT-4o direct prompting on TEMPTABQA-C in LLM-Symbolic Integration. At the item level it yields only a binary right/wrong change, so the audit must beat it on AUROC for each cause separately.

Strongest membership-inference competitor. Min-K% Prob scores how unsurprising the lowest-probability tokens of an item are. Unveiling Causal Reasoning in Large Language Models used it to verify that their data were fresh. It is the standard cheap per-item contamination detector. We found no reported detection AUROC on Lacuna for Min-K% Prob on these benchmarks, so we will measure it ourselves on the same planted models with its threshold tuned on the same held-out split as the audit.

Strongest attribution method. EK-FAC influence attribution, an efficient approximation for tracing which training documents drove a prediction, is the most expensive and most precise competitor. Test of Time reports a 77.5% top-1 hit rate on cloze questions and 17.5% on transformed ones. We run it on a subset of about 200 items because of cost. Beating it means matching its accuracy on verbatim items and exceeding it on register-shifted items, at a small fraction of the compute.

Stale-fact competitor. The fact-duration confidence gate from Mitigating Temporal Misalignment reports a 50–60% reduction in expected calibration error (ECE) on SituatedQA. We adapt it to flag items whose answer depends on an expired fact.

Ablations. The key ablation runs the same audit on bindings extracted from chain-of-thought text by a numeric parser. If prose bindings work as well as program bindings, then programs add nothing. All methods share the same models, items, and tuning split, and every threshold receives the same tuning budget.

Why not just…?

Why not just compare accuracy on perturbed versus original items? This tells you that something changed but not what. A wrong answer on a fact-update item could be a stale-fact leak, a misreading, or an arithmetic slip. Test of Time also shows that aggregate accuracy signals disappear under rephrasing. The planted-cause experiment settles this directly. If the accuracy gap reaches the same per-cause AUROC as the audit, the proposal fails.

Why not just audit chain-of-thought text instead of programs? This is a real threat. Prose reasoning also states numbers, and a good parser might recover them. Prose is ambiguous about which number plays which role, and intermediate values mix with inputs, whereas a syntax tree separates assignments from computation. We run this as the central ablation. If prose-parsed bindings come within a few AUROC points of program bindings, we will report that the diagnostic value comes from making values explicit, not from code. The proposal would then shift to the cheaper prose version.

Why not just use Min-K% Prob or another membership-inference score? These detect familiarity with text, so they should weaken under register shift for the same reason Test of Time's attribution did. They also say nothing about stale facts that appear in no benchmark. The register-shifted, planted-memorization condition tests this. Our prediction is that Min-K% degrades and binding audits do not, because a memorized answer key reappears as a literal regardless of wording.

Decisive Checks

The first check is per-cause detection AUROC on the planted OLMo models across all three variant families. The audit must beat both the accuracy gap and Min-K% Prob on answer-key leaks and on stale-fact leaks, and it must keep its advantage on register-shifted items.

The second check tests the crossover prediction:

  • On the memorization-planted model, answer-key leaks should be rarer under program-of-thought than under chain-of-thought, consistent with the 17.21-to-2.96 reduction.
  • On the obsolescence-planted model, stale-fact leaks should be at least as frequent under program-of-thought, with higher confidence on the wrong answers under self-consistency voting.

If programs reduce both kinds of leak, the "immutable hardcoding" concern is unfounded and we report that.

The third check is transfer. We apply the audit thresholds learned on the planted models, unchanged, to closed models such as GPT-4o-mini on unmodified GSM8K and FinQA, and we inspect a sample of flagged items by hand to estimate precision without planted labels.

Throughout, we report what fraction of programs execute, separately for each model size. The audit is only as broad as the model's ability to write runnable code.