Stale or Invented? Checking Update Summaries Against a Changing Story

When a news story develops, earlier facts go out of date. A casualty count of three becomes twelve, a suspect who was "charged" is later "convicted." A summarizer that tells readers what is new has to get two different things right. It must not invent facts, and it must not repeat facts that were true yesterday but have since been replaced. Current hallucination checkers compare a summary sentence against a fixed source text, so they cannot tell these two errors apart. If the source contains both the old and the new report, "three dead" looks supported. If the source is only the old background, "twelve dead" looks invented. The result is that stale facts pass verification and legitimate updates get flagged. This is a large part of why streaming summarizers such as PDSum stay extractive, copying sentences verbatim: their authors explicitly name hallucination risk as the barrier to generating new text.

We propose to treat fact-checking of update summaries as a four-way judgment rather than a two-way one. Each atomic claim in a summary is labeled as one of the following. It may be current, meaning supported by the latest state of the story. It may be stale, meaning it was once supported but has been superseded. It may be background, meaning it is still true but is not new. Or it may be unsupported, meaning it was never reported. We will build a benchmark for this judgment from real article revision histories. We will then test whether a verifier that explicitly tracks which facts replaced which outperforms strong long-context LLM judges. Finally, we will ask whether feeding its verdicts back to a generator yields abstractive update summaries that are both novel and not out of date.

Core Claim

The claim is that separating "stale" from "invented" requires explicit tracking of which facts have been superseded. Neither checking against the latest document nor checking against all documents achieves it. Our proposed mechanism is a fact ledger. The ledger is a running table of atomic facts, each keyed by an entity and an attribute (for example, "fire / acres burned") and stamped with the time it was reported. When a newer report fills the same slot, the old value is marked superseded. A summary claim is verified by first matching it to a slot in the ledger, and only then checking entailment against that slot's current and past values.

The claim is falsifiable, and the program is designed so that a negative answer is still useful. If an LLM judge given timestamped documents and decomposed yes/no questions separates stale from invented as well as the ledger does, then the ledger is unnecessary. In that case the contribution reduces to the benchmark and the finding that the four-way framing, rather than the architecture, is what matters.

Where this sits

Three strands of prior work motivate the gap, and none of them addresses it directly.

The first strand shows that current checkers break on time. FACTOID trains a multi-task factual entailment model that raises accuracy by 40% over standard entailment. Even so, it reaches only 59% accuracy on its "Temporal Issues" category. TRUE finds that every consistency metric it evaluates degrades once the source text exceeds about 200 tokens, which is far shorter than a multi-version story.

The second strand shows that revision histories carry the signal we need. NewsEdits aligns sentences across 4.6M article versions. NewsEdits 2.0 labels why each edit was made, distinguishing factual, style, and narrative edits. It also uses predicted update likelihood so that GPT-4 declines to answer when facts are still in flux. That use is a gate on answering. It does not verify generated summaries, and it does not say which old claim a new one replaced.

The third strand shows that abstractive update generation hallucinates when left unchecked. Updated Headline Generation produces single-sentence updated headlines with BART from explicitly marked edits. CrisisLTLSum reports BART timeline summaries that invent details as timelines grow. DTELS scores factuality over extracted event facts but never asks whether a later event invalidates an earlier one.

The method draws on two further papers. Structured Discourse Representation for Factual Consistency Verification supplies atomic-fact extraction that keeps time and place qualifiers, which a slot key needs. Ask, Assess, and Refine supplies the loop that turns verifier scores into revision instructions. It also constrains our design, because it shows that unassisted self-correction collapses QAMPARI correctness recall from 18.6 to 4.8. A-MAC, the memory filter in Adaptive Memory Admission Control, is the closest relative of the ledger. It decides what an agent should store using confidence, novelty, and recency scores. It never records that one stored item replaced another, and that replacement link is exactly the distinction we need.

Research Program

The benchmark. We will build a benchmark of four-way labeled claims, called STALE-or-NEW. Its source is NewsEdits revision pairs that NewsEdits 2.0 labels as factual edits, for example "three people died" becoming "twelve people died." From each pair and its surrounding article versions, we will derive claims of four kinds:

  • a current claim stating the new value,
  • a stale claim stating the old value,
  • a background claim stating an unchanged fact from an earlier version,
  • an unsupported claim, created by perturbing the value to one that never appears in any version.

The perturbations will copy FACTOID's hallucination types so that results are comparable. Construction gives us labels for free, but they will be noisy. We will therefore have annotators verify a stratified sample, targeting several thousand items, and report agreement.

A second split will test whether the task survives the move from one article to many. It will use CrisisLTLSum timelines, where later tweets supersede earlier ones, and multi-article topics from DTELS-Bench. Here the latest document usually does not restate still-valid background. This is the regime where simple fixes should break.

The verifier. The verifier has two stages. First, it extracts atomic facts from each document version in time order, using the Structured Discourse extractor. It inserts them into the ledger, and when a new fact matches an existing entity–attribute slot and changes its value, the old entry is marked superseded. Second, it verifies a summary claim. The claim is matched to candidate slots, and entailment is checked only against the short set of facts in those slots. This keeps each premise under the 200-token range where TRUE shows checkers still work. The four-way label then follows from which ledger entry, current or superseded, entails the claim.

The generator loop. An LLM drafts an update summary from the prior summary and the new documents. The verifier then labels each claim, and the stale and unsupported ones are turned into instructions in the A²R style, such as "the acreage figure was revised to X at time t; replace it." We will use at most two refinement rounds, matching A²R's setting.

Baselines

The naive reference point is a verifier that labels a claim current whenever an off-the-shelf entailment model (RoBERTa trained on MNLI) finds it entailed by the concatenation of all versions, and unsupported otherwise. This checker cannot produce stale at all, which quantifies the blind spot.

Two version-aware heuristics come next, and together they make up the most dangerous simple competitor. Latest-only checks entailment against the newest version alone. Diff-then-check labels a claim entailed by an older version but not by the newest as stale. Neither needs a ledger. On single-article revisions, this pair may do very well.

The strongest existing method is a decomposed LLM judge in the BINEVAL style from Ask, Don't Judge. A frontier LLM receives every version with its timestamp and answers atomic yes/no questions per claim, such as "Is this claim stated in any version?" and "Is it contradicted or revised by a later version?" We also run a holistic 1–5 G-Eval-style judge for completeness. BINEVAL is the strongest competitor because it already beats holistic judges on factuality: 0.620 Spearman correlation with human judgments on QAGS and 0.563 on SummEval, versus 0.514 for G-Eval. It also sees the timestamps the ledger uses. No reported result was found on Lacuna for any method on stale-versus-invented classification. FACTOID's 59% on temporal errors is the closest published reference point.

For a fair comparison, all verifiers see identical documents and timestamps. The LLM judges and the ledger's extraction step use the same underlying LLM with the same token budget per claim. Every method gets the same prompt- or threshold-tuning budget on the development split.

One ablation isolates the key ingredient. It keeps the ledger's extraction and short-premise checking but removes the superseded links, checking the claim against all facts in the matched slots regardless of time. This separates the benefit of short premises, which addresses the length problem TRUE identified, from the benefit of tracking supersession.

Beating the strongest baseline means a statistically significant gain in macro-F1 over the four classes. It also requires a gain in the F1 of the stale class specifically on the multi-document split, measured with bootstrap confidence intervals. For generation, the reference numbers are CrisisLTLSum's fine-tuned BART at 36.90 ROUGE-L in its oracle-timeline setting and PDSum's extractive output. PDSum reports only relative gains, +30.9% relevance and +22% novelty over its baselines, so we will rerun it ourselves.

Why not just…?

Why not just check against the latest version? On single-article revisions this is a real threat, and we expect it to be competitive there. It fails whenever the latest text omits facts that are still true, which is the normal case in multi-document streams and tweet timelines. In those settings, background claims get flagged as unsupported. It also fails when the latest version keeps phrasing such as "initially reported three dead," which makes a stale claim look current. The single-article versus multi-document comparison settles this directly. If latest-only matches the ledger on both splits, we will report that version-aware premise selection suffices and drop the ledger.

Why not just give a long-context LLM the timestamped documents and ask? This is the most serious alternative. DTELS reports that GPT-4o prompted with the full context loses information from the middle of large document sets. NewsEdits 2.0 found that GPT-4 did not recognize stale facts without external update scores. Neither result tests our exact question, however, so the BINEVAL-style judge is our main baseline rather than a straw man. We will also report accuracy as a function of the number of versions and the distance between the old and new reports, where the ledger is predicted to win. If it does not win there, the ledger is not worth its cost.

Why not just stay extractive? Copying sentences prevents invention but not staleness. An extractive system can select an earlier sentence containing "three dead" after the count was revised. We will measure the stale-claim rate of PDSum on the same streams to test this. Extraction also cannot merge an old context clause with a new value, which is the operation Updated Headline Generation shows journalists' own edits require.

Why not just ask the generator to fix its own summary? A²R's QAMPARI result, where recall fell from 18.6 to 4.8 under self-correction, is direct evidence against unassisted self-refinement on factual accuracy. We include self-refine as a generation baseline with the same number of rounds, so this is tested rather than assumed.

Proposed Experiments

The first experiment is verification on STALE-or-NEW. We report four-way macro-F1 and per-class F1 for every baseline and the ablation, broken down by split (single-article revisions versus multi-document streams) and by the number of intervening versions. The decisive check is the stale-class F1 on the multi-document split. The ledger has to beat both the diff-then-check heuristic and the timestamped BINEVAL judge there. The ablation without supersession links has to lose most of that gain, which would show the gain comes from tracking replacement rather than from shorter premises. Annotator agreement on the verified subset bounds how much of any gap is real.

The second experiment is generation. We produce update summaries over NewsEdits article sequences, CrisisLTLSum timelines, and TAC 2008–2010 update sets with four systems: the extractive PDSum, a zero-shot LLM, LLM plus self-refine, and LLM plus ledger-guided refinement. Human annotators label the claims in a sample of summaries using the same four-way scheme, and each system is scored on four things:

  • its stale-claim rate,
  • its unsupported-claim rate,
  • novelty relative to the prior summary,
  • ROUGE-L against references where they exist.

The decisive result is ledger-guided refinement lowering both the stale and the unsupported rates relative to self-refine and zero-shot prompting without lowering novelty. Its stale rate should also fall below PDSum's, which would show that abstractive generation with this checker is safer than extraction on the error extraction cannot see.