Language models that write code learn a library's interface from a snapshot of the internet. When the library later renames a function or changes its arguments, every model trained before that change learned the same old version. Ask five such models how to call the updated function and they will often give the same wrong answer, with high confidence. That shared mistake defeats the most common safety check for generated code: let the system answer only when several samples or several models agree, and decline when they disagree. This check is called selective prediction, meaning a model is allowed to abstain. We propose that, for code, the correlation between models' errors is not a fixed property of the models. It is largely set by time. It can be predicted from two dates: when the relevant interface last changed, and when each model's training data was collected.
The core idea is simple to state. Agreement among models should count as evidence only if at least some of those models could have seen the current version of the interface. We build an abstention rule that discounts agreement among models whose training cutoffs all predate the relevant interface change, and gives more weight to models trained after it. We then calibrate the rule so that the error rate on accepted outputs stays below a target. We test this against the simpler rules a reviewer would reach for first, including the bare rule "abstain whenever the interface changed after the model's cutoff." Our claim holds only if the combination beats each ingredient on its own.
Core Claim
The evidence that motivates this comes from three findings. First, models fail together. An audit of more than 350 language models found that pairs agree on the same wrong answer about 60% of the time on four-option questions, where chance is 33%, and that shared provider, architecture and size explain 34% to 62% of that agreement (Correlated Errors in Large Language Models). A separate study found that models from the same family have error correlations above 0.7 (Quantifying Correlations of Machine Learning Models). Second, code models fail on recent interface changes. On CODESYNCBENCH, GPT-4o and Claude 3.5 Sonnet often score below 20% on code completion for recently updated APIs (CodeSync). Third, agreement-based abstention for code has a known blind spot. The joint-CRC method clusters sampled programs by their runtime behavior and abstains when the "good" cluster is too small. Its author identifies a "stable" failure mode in which one model emits the same bad program on every sample, so the agreement check passes it (Selective Code Generation under Correctness and Security Risk).
None of these papers connects error correlation to the history of the software. The audits measure correlation on static multiple-choice benchmarks. CodeSync measures single-model accuracy and has no way for the model to abstain. Joint-CRC and its predecessor, Selective Code Generation (Selective Code Generation for Functional Guarantees), treat each problem as an isolated puzzle and sample from a single model. Our hypothesis is that conditional error agreement between two code models rises sharply when the target interface changed after both of their cutoffs, and is much lower when the change falls between their cutoffs. If that holds, the timing of a change is a cheap and auditable signal for deciding when agreement can be trusted.
Research Program
The first step is measurement. For each of the 220 updated APIs in CODESYNCBENCH, which spans pandas, numpy, scipy, tensorflow, torch and flask, we recover the release date of the change from package history. We then assemble a pool of open code models whose training cutoffs are documented on their model cards and spread out over time, plus the closed models evaluated in the CodeSync and TimeMachine-bench papers where we have access to them. For every pair of models we compute the conditional-on-both-wrong agreement statistic of Kim et al. Two programs count as "the same answer" if they produce the same runtime behavior, which is the equivalence used by joint-CRC. We stratify these statistics by where the change date falls: before both cutoffs, between the two cutoffs, or after both. This analysis is a deliverable whether or not the selector works. It tests whether ensemble redundancy in code is mostly temporal or mostly architectural.
The second step is the selector. For a task whose relevant interface changed at time $t$, each model $m$ with training cutoff $c_m$ proposes a program. Programs are grouped by runtime behavior. Each group $y$ receives a support score
$$ s(y) = \sum_m w(t - c_m)\,\mathbb{1}[\text{model } m \text{ proposed a program in group } y], $$
where $w$ is a small monotone function learned on calibration data. The weight is low when the change postdates the model's cutoff ($t > c_m$) and high otherwise. The system outputs the top group if $s(y)$ exceeds a threshold $\lambda$ and abstains otherwise. We set $\lambda$ with conformal risk control, the same calibration procedure joint-CRC uses, so that the error rate on accepted outputs stays below a budget $\alpha$. Here "error" means the program fails execution or still uses the legacy signature.
Dates are not always known. For code without a clean release record, we estimate $t$ in two ways. One is a static check of the code against the installed package version. The other borrows the version-history encoding of Treude et al. (Encoding Version History Context for Better Code Representation), who showed that a code element's past versions and its age improve clone detection. We would use that encoding only as a fallback, not as the main mechanism.
The calibration has a weak point we state plainly. The conformal guarantee assumes calibration and test tasks are interchangeable in distribution, which statisticians call exchangeability. Software drifts over time, so that assumption is questionable. We therefore use two splits. A random split across APIs is where the guarantee formally holds. A temporal split calibrates on older changes and tests on newer ones, and there we report the observed risk only as an empirical result.
Baselines
The naive reference point is to accept every output from the single best model, with no abstention. On CODESYNCBENCH, the reported figure for frontier models on recent updates is often below 20% code-completion success (CodeSync). This sets the risk any abstention method must reduce.
The strongest existing method is joint-CRC run on a single model (Selective Code Generation under Correctness and Security Risk). It samples $K$ programs, clusters them by behavior, and abstains when the good cluster is too small. It is the closest competitor because it already does behavior-based agreement with a calibrated risk bound. On its own benchmark at $\alpha = 0.10$ it reports 0% observed silent failures at 82.6% coverage. We found no reported result on Lacuna for joint-CRC or any selective method on CODESYNCBENCH or TimeMachine-bench. We will rerun it with the same total number of generated programs as our ensemble, the same execution harness, the same calibration split and the same risk budget.
The second strong baseline is uniform cross-model consensus. It is our selector with $w$ fixed to a constant, which amounts to majority voting across model families with a calibrated threshold. We also compare against the Optimal Weight aggregation rule from Beyond Majority Voting. That rule weights each model by its estimated accuracy and reports a 3.36% accuracy gain over majority voting on MMLU. Here we apply it over behavior clusters instead of answer labels.
Three ablations isolate the role of time. The first is the date rule alone: abstain if the change postdates every model's cutoff, and otherwise accept the most recently trained model's answer. The second learns per-model weights from data but without dates. The third compares a size-matched ensemble whose cutoffs are close together against one whose cutoffs are spread out.
We will say we beat the baselines if, at matched risk $\alpha$ on the temporal split, our selector achieves higher coverage than joint-CRC, uniform consensus and the date rule, with confidence intervals bootstrapped over APIs. Beating some of them but not the date rule would mean the ensemble adds nothing, and we would report it as that.
Why not just…?
Why not just abstain whenever the API changed after the model's cutoff? This is the most serious threat, and it may capture most of the effect. We include it as an ablation for that reason. Our expectation is that it abstains too much. Models sometimes handle post-cutoff changes correctly when the surrounding code or the error message shows them the new interface. It also cannot accept anything in cases where every model is out of date. The comparison of coverage at matched risk will settle this. If the date rule matches our selector, the finding that redundancy is temporal still stands, and the practical recommendation becomes the simpler rule.
Why not just run the tests and accept whatever passes? Execution is part of our error definition, but tests are an incomplete check. On TimeMachine-bench, Claude 4 Sonnet reaches 99.0% pass@1 yet only 78.0% prec@1, a measure of whether each edit was actually needed, and GPT-5 reaches 91.0% and 54.2% (TimeMachine-bench). The gap comes partly from workarounds, such as redefining a deprecated constant locally instead of fixing the import, that pass the tests anyway. Separately, in the joint-CRC experiments a test-plus-linter gate still let bad code through 26% of the time. We include "accept if tests pass" as a baseline on TimeMachine-bench and count edits that are not needed as errors.
Why not just retrieve current documentation? CodeSync reports that a retrieval baseline reached only about a 60% hit rate on its benchmark. Retrieval is also complementary to our approach rather than a replacement: it changes what the models see, not whether their agreement can be trusted. We will run every selector with and without retrieved documentation. If retrieval eliminates the gap in correlation between pre-change and post-change tasks, the temporal weighting becomes unnecessary, and our measurements will show it.
Proposed Experiments
The primary experiment uses CODESYNCBENCH's code-completion and error-correction tasks, about 3,300 test cases in total. We report risk–coverage curves, the area under them, and coverage at $\alpha \in \{0.05, 0.10, 0.20\}$, all under both the API split and the temporal split.
The decisive diagnostic comes first, before any selector is built. If conditional error agreement does not rise for changes that postdate both models' cutoffs, compared with changes that fall between their cutoffs, the premise fails and we stop at the measurement paper.
The transfer experiment uses the 100-repository TimeMachine-bench-Verified subset, whose pypi-timemachine control fixes the dependency date for each repository. An output counts as wrong if it fails the tests or makes unnecessary edits. The question is whether temporal weighting chosen on function-level tasks still controls risk on whole-repository migrations.
The last check examines the "stable" failure mode. Among accepted outputs, we measure what fraction of residual errors are high-agreement legacy calls. The prediction is that joint-CRC accepts these and our selector rejects them.
References
- Correlated Errors in Large Language Models
- Quantifying Correlations of Machine Learning Models
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
- Selective Code Generation under Correctness and Security Risk
- Selective Code Generation for Functional Guarantees
- Encoding Version History Context for Better Code Representation
- Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information
- TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks