A language model answering questions about the world is usually wrong for a boring reason: the world changed after the model, or its search index, last looked. Conformal prediction wraps a model so that it outputs a small set of candidate answers guaranteed to contain the right one, say, 90% of the time. The guarantee assumes the calibration questions used to set the threshold look statistically like the questions the model will face later. Every published conformal method for free-form generation tests this on a randomly shuffled static benchmark. Real deployment runs forward in time, so shuffling quietly lets "future" questions into the calibration pool. The failure that matters most over time is also invisible to existing shift detectors: the question looks familiar, the model answers it confidently and consistently, and the answer is out of date.
We propose to calibrate answer sets on staleness directly. The idea is to replay history. At many past dates we freeze the retrieval index. We then ask questions dated weeks to months after each freeze, whose answers are already known, and record how nonconformity scores (numbers measuring how atypical the true answer is relative to the model's samples) grow with the staleness gap. At deployment, a new question's calibration threshold is set only from historical questions that sat at a similar gap and involved entities with similar pre-freeze news activity. This needs no labels from the deployment period, and every calibration example strictly precedes the deployment cutoff. The method gets a coverage guarantee that holds per staleness level rather than only on average, and it applies to open-ended answers.
Core Claim
Under chronological deployment, the dominant source of broken coverage in open-ended question answering is concept shift: the correct answer to an unchanged question moves. Methods that reweight by the shape of the prompt cannot see this. The shift is, however, well predicted by two quantities available without leakage. The first is the time gap between the question date and the freshest knowledge the system has. The second is how "hot" the question's entities were in the news just before that freeze. Conditioning the conformal threshold on these variables with calibration data manufactured by historical replay should restore coverage within each gap bin, where standard split conformal and prompt-embedding reweighting both under-cover. We also expect it to keep sets more informative than blanket conservatism.
The core mechanism is a weighted conformal quantile over replayed calibration examples. Let $s_i$ be the nonconformity score of calibration example $i$. Let $z_i = (g_i, h_i)$ be its staleness gap $g_i$ and pre-freeze entity activity $h_i$, and let $K$ be a kernel over this two-dimensional space. For a test question with features $z$, the threshold is
$$\hat q(z) = \text{Quantile}_{1-\alpha}\Big(\sum_i \tfrac{K(z_i,z)}{\sum_j K(z_j,z) + K(z,z)}\,\delta_{s_i} + \tfrac{K(z,z)}{\sum_j K(z_j,z)+K(z,z)}\,\delta_{\infty}\Big),$$
where $\delta_s$ is a point mass at $s$ and $\alpha$ is the target miscoverage. The validity condition is testable. Given $z$, the score distribution must be stable across historical epochs. We check this by backtesting across eras rather than assuming it.
Research Program
The work has three parts.
Leakage audit. First we quantify how much of the reported coverage of open-ended conformal methods is an artifact of random splitting. Three prior papers motivate this.
- ConU clusters sampled answers and calibrates a threshold that achieves at or above 90% correctness coverage on TriviaQA, CoQA and MedQA. It explicitly assumes exchangeability.
- Conformal Language Modeling calibrates a sample-and-reject stopping rule under the same assumption.
- EMDOT shows through rolling backtests on medical data that random splits hide temporal decay, but it studies only point metrics like AUROC.
We carry the EMDOT backtesting protocol into conformal calibration for generation. The comparison is coverage under random calibration/test splits versus strict chronological splits on the same pool, stratified by staleness gap.
Historical replay and gap-conditional calibration. Second, we build the replay calibration set and the method above on StreamingQA, which supplies 14 years of timestamped WMT news, roughly 11 million documents, and questions carrying both question dates and passage dates.
- Generator. A frozen open-weight LLM with retrieval over an index truncated at a replay date $\tau$.
- Replay questions. For each $\tau$ on a monthly grid ending before the deployment cutoff $T$, we pose questions dated in $(\tau, \tau + 12\text{ months}]$.
- Nonconformity score. Following ConU, we sample $M$ answers, cluster them with a cross-encoder, and score candidates by cluster frequency and semantic distance.
- Fallback label. Following the missing-mass view of CPQ, the set includes an "everything else" label. When the correct answer never appeared among the samples, the set says so honestly rather than listing wrong candidates. We expect this missing mass to grow with gap. CPQ assumes stationarity; here the missing mass becomes the channel through which staleness shows up.
- Entity activity $h$. The count of indexed documents mentioning the question's entities in the three months before $\tau$. It is computed only from pre-freeze text.
Weighted conformal machinery with a regularizer against set explosion is borrowed from DS-CP. DS-CP estimates density ratios between calibration and test prompts with a sentence embedding and a classifier, and it is evaluated only on multiple-choice MMLU subject pairs. We change what the weights are computed on: staleness features instead of prompt embeddings. We also change the output space to open-ended answers.
Stress tests. Third, we test whether the conditional-stability assumption holds and what happens when it breaks.
- Stability across eras. Calibrate on 2007–2013 replays and test on 2016–2020.
- Abrupt shocks. Include periods where the news mix shifts sharply. EMDOT's coding-change shocks are the analog we have in mind.
- Second domain. Port the protocol to TimeSage-EV, whose rolling releases from February 2023 to May 2026 already enforce cutoff isolation for open-ended analytical answers.
In TimeSage-EV, correctness is judged by keypoint matching from an LLM judge. That judge introduces its own bias, so we treat this domain as secondary evidence.
Baselines
Naive reference: split conformal with random calibration. This is ConU's pipeline calibrated on a random draw from the same pool as the test questions. It represents current practice, and its reported number is the figure to deflate: correctness coverage "consistently at or above 90%" at $\alpha = 0.1$ (ConU).
Chronological split conformal. The same pipeline calibrated on all questions before $T$. It is the honest version of current practice. Beating it means restoring per-gap-bin coverage to within one point of 90% where it falls short.
Sliding-window calibration. Following EMDOT's recent-window training regime, this uses only the last $W$ months before $T$ for calibration. It is the natural recency fix. We tune $W$ on the same pre-$T$ validation replays our method uses.
DS-CP on prompt embeddings. The strongest covariate-shift competitor. DS-CP needs a batch of unlabeled test prompts, so we grant it the full monthly batch of deployment questions, which is an advantage our method does not get. No numeric coverage result is reported on Lacuna for DS-CP, only that it "consistently restored coverage to the target level" on MMLU.
PACE with delayed feedback. The strongest online method. PACE combines reactive threshold updates with a proactive Mahalanobis-distance signal on query embeddings. It reports 2.5x faster recovery and up to 60% lower coverage RMSE than DtACI on simulated streams. We run it on the StreamingQA stream with label delays of 0, 7 and 30 days, since real answers resolve late.
Plain adaptive conformal inference. ACI adjusts the target error level online after each observed miss. It runs under the same delays.
Fairness conditions. All methods share one generator, one index policy, $M$ samples per question, one cross-encoder and one nonconformity score. They differ only in how the threshold is set. Hyperparameters are tuned on the same pre-$T$ replays, with the same number of configurations each.
Ablations isolating the staleness ingredient.
- Condition on gap $g$ alone.
- Condition on entity activity $h$ alone.
- Replace $z$ with the prompt embedding while keeping replay data. This separates the value of replay from the value of the staleness features.
- Drop the "everything else" label.
What counts as beating the baselines. Worst-gap-bin coverage at or above $1-\alpha$ minus one point while every baseline falls below it. Secondary criteria are equal or smaller average set size and "everything else" fraction. For reference, CPQ reports a 16.5% "everything else" fraction on GSM8K at 95% nominal coverage, versus 70.4% for CLM. There is no reported result on Lacuna for any conformal metric on StreamingQA.
Why not just…?
Why not just run adaptive conformal inference or PACE online? They are genuine threats when feedback is fast. Both correct the threshold only after errors, or after a prompt looks unfamiliar. A stale fact asked in familiar words produces neither signal until the delayed label arrives. ACI's own analysis assumes the label is observed before the next step. Our delayed-feedback runs settle this directly. If PACE with 30-day delay matches our worst-bin coverage, the replay idea adds little. We also test the composition, our threshold as the base with ACI corrections on top, because the two are complementary rather than exclusive.
Why not just calibrate on a recent sliding window? Recency helps only if recent calibration questions share the test questions' staleness. They do not. A question calibrated last month was answered with last month's index, while deployment questions drift further from the index each day. Sliding windows also shrink $n$, which inflates sets. The sliding-window baseline, tuned over $W$, measures both effects. If it matches us, that shows staleness is not the operative variable, and we would report that.
Why not just refresh the retrieval index? StreamingQA reports that updated retrieval helps adaptation but still falls well short of human performance: F1 around 0.40–0.50 versus 0.74–0.78 for humans. TimeSage-EV finds agents misreading timestamps even with current data. Refreshing shrinks the gap but does not remove it, since there is always ingestion lag. We run a "fresh index, one-week lag" condition. If coverage there is already nominal under plain chronological conformal, our method's value is confined to the stale-index regime, and we would scope the claim accordingly.
Why not just abstain on anything dated after the cutoff? A date rule is the gap-only ablation taken to its extreme. It would waste most post-cutoff questions, whose answers never changed. Entity activity is what distinguishes those, and the gap-only ablation quantifies how much it adds.
Proposed Experiments and Decisive Checks
Audit result. The first result is the size of the gap between random-split and chronological-split coverage for ConU-style and CLM-style calibration on StreamingQA, plotted against staleness gap. If this gap is under two points everywhere, the problem is smaller than claimed and the rest of the program shrinks to a negative result worth publishing.
Main comparison. Per-gap-bin coverage, set size and "everything else" fraction for all baselines on 2016–2020 deployment months, with calibration from strictly earlier replays. Error bars come from resampling months.
Stability check. Train-era versus test-era comparison of score distributions conditional on $z$ using a two-sample test. If conditional stability fails, the guarantee is void. We would then report how far coverage degrades, rather than hide it.
Portability check. The same protocol on TimeSage-EV using its keypoint accuracy as the admission criterion. This checks whether staleness conditioning transfers from factual lookup to the forecasting and attribution tasks. There, frontier models already drop to 53.1 on hard tasks for Gemma-4 per TimeSage-EV.
References
- ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees
- Conformal Language Modeling
- Evaluating Model Performance in Medical Datasets Over Time
- StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
- Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Models
- Domain-Shift-Aware Conformal Prediction for Large Language Models
- TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments
- Beyond Reactivity: Proactive Adaptive Conformal Inference for Online LLM Factuality
- Adaptive Conformal Inference Under Distribution Shift