Version-Aware Replay for Legal Judges When the Law Changes

An automated judge is a language model that grades other models' answers. In law, the correct grade can flip overnight. If a statute is amended, a response that was right last month is now wrong, and a judge fine-tuned on last month's verdicts will keep rewarding it. The usual defense against a fine-tuned model forgetting old skills is experience replay: keep a small memory of past training examples and mix them into every new training batch. For a judge facing changing law, though, replay works against you. The buffer is full of verdicts whose truth has expired, and replaying them teaches the judge the old law again.

We propose to make the replay buffer aware of which version of the law each stored verdict depends on. When a provision is amended, buffered judgments that depend on it are not discarded. They are turned into paired examples: the same student answer is shown once with the old statute text in context and the old verdict, and once with the amended text and the new verdict. Judgments that do not depend on the amendment are replayed as usual. The bet is that these pairs teach the judge to take its verdict from the rule written in front of it rather than from rules stored in its weights. That habit is exactly what frozen retrieval-augmented models lack: LeKUBE reports that they "sometimes prioritize their outdated internal knowledge over the provided context." A judge that learns this deference should stay correct across a sequence of amendments while it is also being refreshed for stronger generators and new question types.

Core Claim

A legal judge continually fine-tuned with version-aware paired replay will (a) issue fewer stale verdicts on amended provisions, (b) keep accuracy on unamended provisions, and (c) keep the shelf-life properties of continual fine-tuning on new generators. It will do better on all three than a frozen judge given retrieved statutes, than standard or surprise-prioritized replay, and than replay that simply deletes affected items. The new ingredient is the old-rule/new-rule contrast built from the judge's own history. Removing either half of the pair should measurably weaken deference to context.

Where This Sits

Three bodies of work motivate the gap. On the Shelf Life of Fine-Tuned LLM-Judges shows that fine-tuned judges age as generators improve. It finds that sequentially fine-tuning an old judge on new data gives more balanced performance than retraining from scratch. Its notion of time, however, is generator capability on math (DeepScaleR) and multiple-choice questions (MMLU-Pro). The ground truth itself never changes, and its continual learning is plain sequential fine-tuning with no buffer. LeKUBE does study changing ground truth: 180 legal amendments written by professionals. It shows that full fine-tuning memorizes amended statutes (near 93% recitation accuracy) but fails to reason with them, and that surgical editors such as ROME and Self-Edit damage unrelated law. It studies models as answerers, though, not as graders. LeMAJ shows that breaking legal answers into atomic "legal data points" and checking each one against source documents raises correctness correlation with experts to 0.700 on a LegalBench subset. Its source documents are fixed, and its judge is a prompted frontier model that is never trained.

Replay methods supply the machinery. Continual Learning with Tiny Episodic Memories shows that even one stored example per class beats regularization methods such as EWC (elastic weight consolidation) and A-GEM (averaged gradient episodic memory) by 7 to 17 points. SuRe selects high-loss sequences for an LLM's buffer and smooths low-rank adapter weights with a slow moving average. Retrospective Adversarial Replay selects samples near the boundary of the incoming task. All of these choose what to replay based on loss or geometry. None asks whether a stored label is still true, and that is the gap we fill. Mitigating Temporal Misalignment by Discarding Outdated Facts does reason about expiry, predicting how long a fact stays valid. It uses that prediction only to trigger retrieval in an answering model. We borrow its stance that staleness should be explicit, but we act on it during training rather than at inference.

Research Program

The setting is a sequence of update rounds. In each round the judge receives new graded data: responses from newer and stronger generators, plus possibly a batch of statutory amendments. Every buffered item stores four things: the question, a student response, a verdict, and the identifiers of the statutory provisions the verdict depends on. Dependencies are recorded when the item is created. For LeKUBE scenarios they come from the annotated provision that each update modifies. For other items, an extraction pass links the cited provisions, and we audit that pass by hand on a sample.

When provision $p$ is amended, each buffered item that depends on $p$ is re-judged under the new text, and two training examples are created from it. Let $x$ be the question and response, $r_{\text{old}}$ and $r_{\text{new}}$ the provision text before and after the amendment, and $y_{\text{old}}$, $y_{\text{new}}$ the verdicts under each. The training loss for a round is

$$ L = L_{\text{new}} + \lambda_u \, \mathbb{E}_{\text{unaffected}}\big[\ell(x, r, y)\big] + \lambda_c \, \mathbb{E}_{\text{affected}}\big[\ell(x, r_{\text{old}}, y_{\text{old}}) + \ell(x, r_{\text{new}}, y_{\text{new}})\big]. $$

Here $\ell$ is the judge's verdict loss, and $L_{\text{new}}$ is that loss on the round's fresh data. The weights $\lambda_u$ and $\lambda_c$ set how much training goes to locality replay and to contrast replay. Whenever the verdicts differ, the pair forces the verdict to depend on the rule text. At deployment the judge always receives the current provision, retrieved from a versioned statute index.

The relabeling is the part where errors can creep in, so we treat it as an object of study in its own right. LeKUBE provides professionally written pre-update and post-update law together with reasoning scenarios. From these we can label generator responses for correctness under each version: a strong model grades each response against the version-specific reference answer, and legal annotators audit a stratified sample of these labels. We will report how often the automatic relabels disagree with the annotators. If they disagree often, the method inherits that noise, and the experiments must show whether it still helps. The judge uses LeMAJ-style atomic tagging, so verdicts are attached to individual claims. This lets an amendment flip a single claim's tag rather than the whole grade.

Baselines

The naive reference point is a judge fine-tuned once on pre-amendment data and never updated. It is evaluated both without statute text and with the current statute retrieved into the prompt. The second variant, a frozen judge with retrieval, is the strongest existing competitor. LeKUBE found retrieval to be the most stable update method, with perfect locality by construction, so any gain we claim must come from the frozen judge's deference failures. We build it with the same backbone and the same versioned index and retriever that our method uses, so that differences come from training alone.

The second baseline is sequential fine-tuning on each round's new data, which is the continual recipe of the shelf-life paper. We run it with SFT followed by DPO (supervised fine-tuning, then direct preference optimization), since that paper found DPO more robust to shift. The third is standard experience replay with a reservoir buffer of matched size, as in Tiny Episodic Memories. It replays stale labels unchanged. The fourth is SuRe, which uses surprise-prioritized selection and the slow moving-average learner on low-rank adapters. It is the strongest replay method for LLMs in the file: it reports up to 5 accuracy points over prior methods and 78.0% average accuracy on the 15-task benchmark. The fifth is discard replay, which deletes every item that depends on an amended provision and keeps the rest.

Two ablations isolate the key ingredient. New-only replay re-labels affected items under the new rule but drops the old-rule half of the pair. One-shot rule-conditioning trains the judge once on synthetic counterfactual rule edits before deployment and then updates it by sequential fine-tuning without contrast replay.

For fairness, all methods share the backbones used in the shelf-life paper (Llama-3.1-8B and Ministral-8B), the adapter rank, the buffer size in examples, the number of gradient steps per round, and an equal hyperparameter search budget. The prompted-judge baselines use the same retrieved context. We have found no reported result on Lacuna for any LLM judge evaluated on LeKUBE amendments, so the numbers to beat will be measured here. As external anchors, LeMAJ's 0.700 correctness correlation on LegalBench is the bar our judge must not fall below on unamended LegalBench items, and the shelf-life paper's refresh gain of up to 7.6 points is the scale of improvement continual updating has delivered so far.

Beating the strongest baseline means two things. First, a statistically significant reduction in stale-verdict rate on amended provisions relative to the frozen retrieval judge. Second, an unamended-provision accuracy no more than one point below it and a shelf-life score on newer generators no worse than sequential fine-tuning.

Why not just…?

Why not just keep the judge frozen and retrieve the current statute? This is the most serious threat, and the proposal is built to test it head-on. LeKUBE's evidence that retrieval-augmented models defer to stale internal knowledge is about answering models, not graders, and it may not carry over. If the frozen retrieval judge already shows a near-zero stale-verdict rate on our amended items, the premise fails. In that case we would report the result as a finding and drop the training component. Note also that a frozen judge cannot be refreshed for stronger generators, which the shelf-life paper shows makes judges age, so retrieval alone does not address half of the problem.

Why not just prompt a frontier model with the new rule? That is a fair baseline, and we include a prompted frontier judge with identical retrieval. The file gives reasons for doubt. Mainstream models score below 50% F1 on predicting appellate reversal in AppealCase, and frontier judges show positional bias in A Benchmark for Long-Form Medical Question Answering. The file does not show that they ignore context on amended law, however. We measure their stale-verdict rate, and if it is low, a small trained judge must justify itself on cost alone.

Why not just delete stale items from the buffer? Deletion is the obvious fix and it is cheap, so it is a baseline. Our hypothesis is that deletion protects locality but teaches nothing about following the rule in context. The judge's weights may still encode the old rule from earlier rounds. The comparison between discard replay and paired replay, read on stale-verdict rate, decides this.

Why not train rule-following once, up front? If a single round of counterfactual rule-edit training makes the judge defer to context permanently, continual contrast replay is unnecessary. We suspect later rounds of fine-tuning on new generator data erode this deference, but that is an empirical claim. The one-shot rule-conditioning ablation, tracked round by round, measures whether deference decays.

Surgical editing is excluded as a primary competitor because LeKUBE already shows it often yields negative aggregate scores. We run it once for completeness.

Proposed Experiments

The main benchmark is a judge version of LeKUBE. We sample responses from weak and strong generators (the generator tiers used in the shelf-life study) to LeKUBE's recitation, reasoning, and locality questions. We label each response under the pre- and post-update law, and release the amendments over five or six sequential rounds, with generator strength also increasing across rounds. Unamended LegalBench rule-conclusion and rule-application tasks, where the rule is given in context, serve as a second locality check and as the comparison point to LeMAJ.

We report five measures, each against human-audited labels. Stale-verdict rate is the fraction of amended items where the judge's verdict matches the old law. Updated verdict accuracy measures correctness under the new law. Locality accuracy covers untouched provisions. The shelf-life paper's future-proofing and backward-compatibility scores cover generator shift. Correlation with expert grades is computed on the audited subset.

The decisive checks are three. The first is whether the frozen retrieval judge actually issues stale verdicts at a non-trivial rate; this tests the premise. The second is whether paired replay beats both discard replay and new-only replay on stale-verdict rate without losing locality; this tests the mechanism. The third is whether the one-shot rule-conditioned judge's deference decays across rounds while the paired-replay judge's does not; this tests why continual contrast is needed at all. A secondary analysis checks for an error-compounding effect. Rounds with more amendments per provision should widen the gap between plain replay and paired replay, if stale labels in the buffer compound with each round.