Counterfactual Likelihood Tests for Superseded Facts in Long Dialogues

Chat assistants that remember long conversations keep making one specific mistake. A user says their budget is $600, later says it is now $1200, and the assistant still plans around $600. The obvious explanation is that the memory system failed to retrieve the update. That explanation is wrong. Can Agent Memory Systems Track Evolving State? shows the error, which it calls state drift, persists even when every relevant turn is placed in the prompt. Models anchor on whichever value is louder or more often repeated. Adding step-by-step reasoning makes this worse, because the model uses the extra steps to justify the stale value. The failure therefore happens when the model reads the context, not when the system retrieves it. A natural fix is to find superseded statements and mark or remove them before the reader ever sees them.

This note proposes doing that with a likelihood test instead of an LLM parser. For each earlier statement, we ask a small language model one question: does reading a particular later turn make the earlier value less probable as the current value? If it drops sharply, the later turn has superseded it. The test uses only forward passes, needs no labels, and normalizes itself against the model's own baseline belief. That self-normalization is the move that made likelihood-ratio methods work in out-of-distribution detection. The same scorer also lets us measure a second effect: how much the model's pretrained world knowledge pulls it back toward outdated values, which is the knowledge-obsolescence side of the problem.

Core Claim

The obvious way to connect anomaly scoring to dialogue memory is to fit a density model over the conversation and flag low-likelihood turns. That approach is aimed at the wrong target. An update ("budget is now $1200") is not unusual in embedding space. It sits right next to the statement it replaces, because it is about the same attribute. Density over turns measures topical surprise. Topical surprise mostly picks up harmless rare chatter, the "rarity is not anomaly" problem documented in Search Problems in the Domain of Multiplication. It does not pick up changes in which value is true.

The quantity that matters is relational: how one later turn changes the probability of an earlier claim. Let $s_i$ be an earlier turn, $c_i$ a value span it asserts (a number, entity, or date), and $s_j$ a later turn. Let $q$ be a fixed probe such as "The current value is:". We define the supersession score as

$$S(i,j) = \frac{1}{|c_i|}\Big[\log p_\theta(c_i \mid s_i, s_j, q) - \log p_\theta(c_i \mid s_i, q)\Big],$$

where $p_\theta$ is a frozen scoring language model and $|c_i|$ is the number of tokens in the span. Strongly negative $S(i,j)$ means turn $j$ makes value $c_i$ no longer current. The subtracted term plays the role of the background distribution in FLatS, which scores intent queries by in-distribution minus background feature energy. Subtracting it cancels the length and token-frequency biases that make raw likelihood unreliable, as shown in Towards Textual Out-of-Domain Detection without In-Domain Labels. A turn is marked stale when its minimum score over later turns falls below a threshold. That threshold is set once on held-out synthetic dialogues.

The claim is that this pre-reading pass removes most state-drift errors at a small fraction of the ingestion cost of LLM-parsed state graphs. We also claim that the score is responsible for the gain, not the act of pruning itself.

Research Program

The system has three stages.

Candidate pairing. Each turn's value spans are found with a light tagger for numbers, entities, and dates. Whole-turn scoring is the fallback when no span is found. Each span is paired with its top-k most similar later turns under a sentence encoder. Here semantic similarity is used deliberately: updates resemble what they replace. This inverts the usual retrieval role and keeps the number of pairs close to linear in conversation length.

Scoring. A 1–8B model computes $S(i,j)$ for every pair. This requires two forward passes per pair and no generation.

Rewriting. Stale turns are either deleted or annotated inline, for example with "[superseded at turn j]". The cleaned history then goes to an unchanged reader model. We compare deletion and annotation because deletion risks losing context the reader still needs.

Derived values. Because $p_\theta$ conditions on whole turns, a derived value can also lose probability once its base fact changes. An example is a total computed from the old budget. If this works, the scorer reproduces part of what StateMem's deterministic Rechecker does through explicit dependency edges. We test this directly on StateMemBench's derived-value cases rather than assuming it.

Parametric prior. The same model gives a prior $\log p_\theta(c \mid q')$, where $q'$ is a context-free probe naming the attribute. We hypothesize that drift is most severe when the stale value has a higher parametric prior than the update. An example is a user telling the assistant about a post-cutoff change in who holds an office. This links the salience trap to the "update fallacy" in SituatedQA, where models given updated corpora remain 15 points less accurate on changed facts than on stable ones. It also differs from Mitigating Temporal Misalignment by Discarding Outdated Facts. That work predicts how long an isolated fact stays true. We score observed supersession inside a conversation, and use the prior only as a diagnostic and a possible extra feature.

Positioning Against Closest Work

Four memory systems sit closest to this proposal.

StateMem parses turns into state units with dependency graphs, using LLM calls at ingestion. LeanMem routes segments into profile, event, and record memories using rules and LLM calls. ACR trains a discrete router that picks context-rewriting operators under teacher supervision. LongMemEval reads memory through structured Chain-of-Note. All four decide staleness with discrete parsing, rules, or a supervised classifier. None scores supersession as a continuous, label-free likelihood contrast.

On the anomaly side, FLatS and the textual out-of-domain detection work supply the ratio idea, but both score single utterances in isolation. Applying them per turn is the naive design this proposal rejects.

Two further results constrain the design. Do LLMs Benefit From Their Own Words? shows that omitting past assistant turns preserves quality while shrinking context up to tenfold. Any pruning method must therefore beat that cheap heuristic. The finding that models fail to resolve all dependencies even with ideal evidence, from Don't Scroll Back, warns that cleaning the context cannot remove reader errors that are not about staleness. We expect gains concentrated on drift-attributed errors, not on all errors.

Baselines

Naive reference. The first reference point is full-context reading with the same reader and no pruning. A second, cheap reference is assistant-turn omission as in Do LLMs Benefit From Their Own Words?. Both run on identical histories and reader models. Our method must beat the better of the two on current-state accuracy, and omission must be combined with our method to check that the two are complementary.

StateMemWrapper. The strongest low-cost competitor is the single-call "trace-then-resolve" prompt from Can Agent Memory Systems Track Evolving State?. The paper reports lifts of +32 to +67 points across six memory backends. It is the strongest competitor because it targets exactly the reading-time failure we target, without any ingestion cost.

StateMem. The full StateMem system reports current-state accuracy of 0.363 versus 0.205 for the strongest prior baseline on StateMemBench with a DeepSeek-V4-Flash backbone. That 0.363 is the number to beat.

Running the comparison fairly. We run StateMem, StateMemWrapper, and our pruning with the same reader and memory backends. We report ingestion tokens and wall-clock time alongside accuracy. Beating them means either higher current-state accuracy than 0.363 at matched reader, or statistically equal accuracy at substantially lower ingestion cost. We also test whether our pruning stacks with the wrapper.

LongMemEval. The relevant reference is LeanMem with GPT-4.1-mini, which is 15.1 points above SimpleMem on LongMemEval-S. The file gives no reported per-task number for the knowledge-update subset, so we will re-run LeanMem ourselves on that subset under matched backbone.

Ablations. Three ablations isolate the key ingredient. The first replaces $S(i,j)$ with raw value likelihood under the full context, with no subtraction. The second replaces it with a per-turn Gaussian mixture density score in the style of the textual out-of-domain detection paper. The third prunes all but the most recent mention of each tagged attribute, a pure recency rule. All use identical pairing, threshold calibration, and rewriting.

Why not just…?

Why not just keep the latest mention of each attribute? Recency fails in three cases. It fails when "the latest mention" is the user restating an old value conversationally ("back when it was $600"). It fails when attributes are multi-valued ("I also want a flight on Friday"). It fails when the change is implicit in a derived quantity. Our recency ablation runs on exactly these StateMemBench cases. If recency matches the likelihood score overall, we will report that the benchmark under-represents these cases. We will then build a targeted split rather than claim a win.

Why not just use an off-the-shelf contradiction classifier on sentence pairs? This is the most serious threat, and we treat it as a co-equal baseline using a natural-language-inference cross-encoder with the same pairing and threshold procedure. Supersession is not the same as contradiction. "We bumped it up" contradicts nothing in isolation, and its meaning depends on surrounding context that a pair classifier never sees. Still, a contradiction classifier may do well on explicit updates. The decisive comparison is on implicit and derived updates. If the classifier ties there too, the proposal's contribution narrows to the claim that reading-time pruning fixes drift, with the scorer as an interchangeable part. We would say so.

Why not just let a long-context model or chain-of-thought sort it out? The file already answers this. LongMemEval reports 30–60% declines for long-context models relative to oracle settings. StateMem's authors show drift survives oracle retrieval and that reasoning traces can lower accuracy. We include a chain-of-thought reader arm only to confirm that pruning helps it as well.

Why not just fit a density model over turns, the obvious anomaly-detection move? Updates are near their predecessors in feature space, so per-turn density should miss them and flag benign rare chatter instead. The Gaussian mixture ablation tests this prediction directly.

Proposed Experiments

Main evaluation. StateMemBench is the primary benchmark because its closed-pool grading attributes each error specifically to picking a superseded value. We report current-state accuracy and the drift-attributed error rate. The LongMemEval_S knowledge-update and temporal-reasoning subsets test transfer to more natural conversations. LongMemEval_M, at about 1.5M tokens, tests whether the near-linear pairing keeps scoring cost manageable.

Controlled parametric-prior set. We build a controlled set by turning SituatedQA facts with time-dependent answers into dialogues. In each, a user first states one value and later states another. We vary two things: which value matches the scorer's parametric prior, and how many times the stale value is repeated.

Decisive checks. Four checks decide whether the idea holds.

The first is oracle-context recovery. With all relevant turns supplied, pruning should recover most of the accuracy gap between full context and a hand-cleaned history. If it does not, the scorer is the bottleneck.

The second is the salience curve. Drift rate as a function of stale-value repetitions should flatten under pruning while rising without it.

The third is the prior-gap prediction. Drift rate should rise with the prior log-probability of the stale value minus that of the update. Pruning should help most where that gap is largest. If drift does not depend on the prior, the obsolescence link is falsified and we drop it.

The fourth is precision. We measure how often non-superseded turns are pruned, with human labels on 500 sampled pairs. We also measure the downstream cost of false prunes by comparing the deletion and annotation variants.

Success criterion. Success means beating StateMemWrapper on drift-attributed errors and matching or exceeding StateMem's 0.363 at lower ingestion cost. The ratio score must also clearly outperform the raw-likelihood, density, and recency ablations.