Keeping a Retrieval-Augmented Judge Honest When Its Index Goes Stale

Language models are now routinely used as graders: an "LLM judge" reads a question and a candidate answer and decides whether the answer is correct. The most careful versions of this idea come with a statistical promise. They accept a verdict only when the judge is confident, look up evidence when it is not, and send the hardest cases to humans. Thresholds are tuned so that the fraction of accepted verdicts that are wrong stays below a target level. That promise is only as good as the evidence store the judge consults. Real evidence stores are refreshed on a schedule: nightly, weekly, or whenever someone gets around to it. Between refreshes, the facts that matter most, the ones that just changed, are exactly the ones the store gets wrong. A judge that checks a correct, up-to-date answer against a stale passage will confidently mark it wrong.

We propose to measure this failure and to repair the guarantee. We make the age of the index an explicit, observable variable. The judge's acceptance thresholds are calibrated separately for each index age. A cheap staleness signal comes from disagreement between keyword search and embedding search, because keyword search picks up newly coined names and dates that embeddings smooth over. The practical output is a curve an operator can act on: for a target error rate, how much of the evaluation workload the judge can handle automatically as a function of how often the index is refreshed. This turns "how often should we update the database?" into a question with a measurable answer.

Core Claim

Judge, Retrieve, or Abstain supplies the method we build on. It routes each verdict through a judge that relies on its own knowledge, then a retrieval-augmented judge, then abstention. Entropy thresholds are calibrated with Clopper–Pearson bounds so that the false discovery rate (FDR, the share of accepted verdicts that are wrong) stays at or below $\alpha$. It reports large coverage gains from retrieval: from 7% to 82% on NQ-Open with a Qwen3-8B judge at $\alpha = 0.20$, and from 40% to 85% on HotpotQA with a Llama-8B judge. It treats retrieval as a black-box web search. It checked drift only as URL turnover over three months, on questions whose answers do not change.

Our claim is twofold. First, on time-sensitive questions this calibration silently breaks once the index lags the world. We expect the breakage to be worst for answers that are correct now but contradicted by the most recent indexed version of the fact. Second, stratifying calibration by index lag restores the guarantee at each lag. Adding a sparse–dense disagreement feature recovers much of the coverage that lag-stratification alone gives up. Formally, for index lag $\ell$ (time between the last index refresh and the moment the judged answer is asserted) we target

$$\mathrm{FDR}(\ell) = \Pr(\text{verdict wrong} \mid \text{verdict accepted},\ \text{lag} = \ell) \le \alpha \quad \text{for every lag bucket } \ell,$$

which is valid because $\ell$ is known at test time. Both the index build time and the query time are logged, so per-bucket calibration is ordinary stratified conformal calibration and needs no shift correction.

Three lines of work motivate this. ChronoQA builds a temporal question-answering benchmark from about 300,000 Chinese news articles spanning 2019–2024. It finds that 72% of end-to-end errors come from retrieval failing to find temporally aligned evidence, and that implicit temporal expressions account for 69% of analyzed failures. LeKUBE benchmarks knowledge-update methods on amended Chinese law and finds that models sometimes prefer outdated internal knowledge over retrieved new statutes. That is precisely the conflict a judge faces. MRAG shows that ordinary retrievers are "date-blind" and that splitting a query into content and time constraints improves gold-evidence recall by 11%. None of these involve a judge or a risk guarantee. The judge paper involves no changing corpus.

Research Program

We first build a replayable, time-stamped evidence store. ChronoQA's article collection is dated, so we can rebuild the index as it would have existed at any cutoff date and pose each question at a later "assertion date." We build three indexes for every cutoff. The first is a keyword index (BM25, which scores documents by weighted word overlap). The second is a dense index (a dual encoder that maps text to vectors and retrieves by similarity). The third is a hybrid that combines the two by linear score interpolation, in the style analyzed by Luan et al..

We then generate candidate answers with several generator models and label each answer's correctness as of the assertion date using ChronoQA's gold answers. LeKUBE supplies a second domain with clean before/after versions of each statute. There, staleness means "the index still holds only the old article."

On this testbed we run the Judge, Retrieve, or Abstain pipeline unchanged. It is calibrated on a fresh index, and we measure empirical FDR and coverage as lag grows from zero to months. This is the measurement half. We expect FDR violations concentrated on questions whose answer changed inside the lag window.

The repair half has two parts. The first is lag-stratified calibration: separate threshold pairs for each lag bucket, fitted on calibration questions replayed at that lag. The second adds a staleness feature to the routing rule. The feature measures disagreement between the keyword and dense retrievers' top results, along with the spread of dates among retrieved passages. The rationale comes from the hybrid-retrieval literature. Dense vectors blur rare strings such as new identifiers, while keyword matching preserves them, as Luan et al. note. So when the world has moved, the two retrievers should diverge more than usual. This feature is cheap: it needs no extra judge calls.

The deliverable is a family of coverage-versus-lag curves at fixed $\alpha$. They play the role that the age-of-information analysis of Update or Wait plays for sensor networks, which shows that freshness should be scheduled rather than maximized. Here the curves let an operator pick a refresh interval for a required coverage.

Baselines

The naive reference point is a judge that uses only its internal knowledge with calibrated thresholds, which is Mode 1 of Judge, Retrieve, or Abstain. It cannot be hurt by index lag, but its reported coverage is low (7% on NQ-Open at $\alpha = 0.20$). On time-sensitive questions it should also suffer from its own training cutoff.

The strongest existing method is the full Judge, Retrieve, or Abstain pipeline, calibrated once on a fresh index and then deployed as the index ages. Its reported results are 82% coverage on NQ-Open and 85% on HotpotQA at $\alpha = 0.20$ with FDR at or below target. We found no reported result on Lacuna for any retrieval-augmented judge on ChronoQA, LeKUBE, or TEMPRAGEVAL, so we will establish those numbers ourselves. We run it with the same judge models (Qwen3-8B and Llama-8B), the same calibration set sizes, the same candidate answers, and the same retrievers as our method. The only differences are the calibration stratification and the added feature. Beating it means two things. Our method must keep empirical FDR at or below $\alpha$ in every lag bucket where the baseline exceeds it. At lags where both are valid, its coverage must be within a few points of the baseline's.

Two ablations isolate the ingredients. Lag-stratified calibration without the disagreement feature tells us whether stratification alone restores validity, and at what coverage cost. Replacing the hybrid retriever with BM25-only and dense-only retrievers tells us whether the hybrid matters for the judge, and whether the disagreement signal is informative beyond what either retriever offers alone.

A third comparison places MRAG's semantic–temporal reranking in front of the fresh-calibrated judge. This tests whether better temporal retrieval alone fixes the problem. MRAG reports a 9.3% gain in top-1 answer recall on TEMPRAGEVAL, but no result found on Lacuna for it as a judge front end.

Why not just…?

Why not just refresh the index more often? This is the most serious alternative, and it is partly what our coverage-versus-lag curves are for. If FDR violations vanish at a refresh interval operators can afford, the honest conclusion is "refresh daily," and the repair half becomes unnecessary for that domain. Two things argue that refreshing is not sufficient. First, some lag always remains: news and legal amendments arrive continuously, and AURORA motivates continual indexing precisely because full rebuilds lag streaming data. Second, freshness does not remove stale passages. Old and new versions of a fact coexist in the index, and LeKUBE shows models can still side with the outdated version. Our zero-lag condition directly tests whether conflicts persist even with a perfectly fresh index.

Why not just recalibrate the thresholds periodically on new labeled data? Periodic recalibration on fresh labels is valid but needs a continuous stream of human correctness labels. Removing that labor is the point of an automated judge. Lag-stratified calibration is fitted once, by replaying historical snapshots whose labels already exist. We will test whether thresholds fitted on 2019–2022 replays remain valid on 2023–2024 questions. If they do not transfer, periodic labeled recalibration wins, and we will report that.

Why not just put a date-aware retriever in front of the judge? Temporal reranking such as MRAG reduces date-blindness, but it reranks only what the index contains. If the new fact has not been ingested, no reranking can surface it, and the judge will still be confidently wrong. The MRAG-front-end baseline measures how much of the FDR inflation is a ranking problem and how much is a coverage-of-the-index problem. We expect ranking to help at short lags and stop helping once the changed fact falls inside the lag window.

Proposed Experiments

The first decisive check is whether the problem exists. On ChronoQA replays with lags of zero, one week, one month, and three months, does the fresh-calibrated judge exceed $\alpha = 0.20$ on questions whose answer changed within the lag? We will compare against NQ-Open and HotpotQA as static controls, where FDR should stay at target. If violations do not appear, the proposal reduces to a negative result worth reporting: the judge's guarantee is robust to staleness.

The second check is whether lag-stratified calibration holds FDR at or below $\alpha$ in every bucket. It must do so on ChronoQA and on LeKUBE's old-statute versus new-statute indexes, and its coverage cost must be reported against the baseline.

The third check is whether the sparse–dense disagreement feature recovers coverage without breaking validity. We will compare hybrid, BM25-only, and dense-only retrievers under the same lag buckets. TEMPRAGEVAL is a secondary test of whether the effect survives perturbed temporal constraints.

Throughout, the headline figure is the coverage-at-fixed-FDR curve as a function of refresh interval, reported for each retriever.