Retraction-Aware Rollback for Streaming Literature Agents

AI systems are starting to read the scientific literature as a stream. They keep a running summary of a field, update it every few weeks as new papers arrive, and use it to propose hypotheses. These systems have the same weakness as an online object tracker that updates itself from its own predictions. A wrong input gets absorbed into the running state, later updates build on it, and the error compounds. The scientific record makes this worse in a specific way. A flawed paper is often corrected or retracted months or years after publication, and by then the agent has already "metabolized" it into derived claims, strengthened-evidence notes, and hypotheses. Deleting the retracted paper afterwards does not remove those echoes.

We propose to give a streaming literature agent claim-level provenance, meaning a record of which source papers and which earlier claims each statement in its running state rests on. When a retraction or erratum arrives, the agent then performs a cascading rollback that removes or downgrades exactly the dependent content. Separately, newly published claims sit in a provisional tier until independent work corroborates them, which limits how far an error can spread during the window before it is corrected. The central measurable question is whether surgical rollback can match the gold standard, which is re-running the whole stream with the bad paper removed, at a fraction of the cost and with far less disturbance to unrelated content.

Core Claim

Streaming literature agents accumulate residual contamination: content that survives the deletion of a retracted source because it was derived from that source in earlier update steps. Provenance-tracked cascading rollback should eliminate most of this residue. It should come close to a full counterfactual replay in both contamination removal and downstream hypothesis quality, while costing much less and rewriting far less unrelated content. A second claim is that a two-tier state reduces contamination during the pre-retraction window without destroying the lead time that makes streaming agents useful. The tiers are a corroborated "anchor" tier and a provisional "recent" tier, in the spirit of dual-template trackers.

The motivating system is Continuous Knowledge Metabolism (CKM). CKM updates Markdown topic files over 2-month windows through "contrastive updates": it appends claims, strengthens evidence, and flags contradictions. On a 50-topic arXiv benchmark it reports a 399-day median lead time and validated hypotheses for 72% of topics. CKM ingests every paper unconditionally and has no way to retract a claim once it is in the state. Its update loop is exactly the first-order, self-feeding update that the tracking literature identifies as the cause of drift (A Survey of Appearance Models in Visual Object Tracking). We build on CKM's pipeline and add the missing correction machinery.

Why tracking is the right lens, and where the analogy stops

Three tracking ideas transfer directly. DLST pairs an immutable first-frame template with an evolving one, so that a polluted dynamic template cannot overwrite the trusted anchor. This motivates our anchor and provisional tiers. KeepTrack explicitly keeps the identities of distractors, not just the target, and uses confidence-based memory replacement so that distractor features never enter its update buffer. This motivates keeping retracted claims in the state as tagged "do-not-synthesize" entries rather than silently deleting them, so that a later paper restating the same finding can be matched to them. SoTTA shows that filtering what enters the adaptation buffer matters enormously under noisy streams. It reports 80.0% accuracy on noisy CIFAR10-C versus 52.1% for TENT, a test-time adaptation method that updates on every incoming sample. That result motivates gating at ingestion.

The analogy breaks in one important place. A tracker never learns, frames later, that frame 40 was wrong. The scientific record does, through retraction notices and errata. This delayed ground-truth signal is what makes rollback possible and worth studying, and it has no counterpart in the tracking work.

RSI for Science constrains the design. It argues that self-evaluation drifts upward while true validity falls, and that systems need external disconfirmation gates and path-sensitive metrics such as maximum drawdown. Retraction notices are a natural external disconfirmation signal, and we adopt drawdown as a stability metric. That paper is a position and reporting framework with synthetic vignettes and no literature engine. AutoSupervision supplies a lesson we reuse in evaluation. Models asked whether a revision fixed a concern are fooled by the author's response letter (verification peaks around 0.501), and checking requires grounding in the actual manuscript body. Analogously, we check the agent's state itself after a correction rather than trusting its own report that it "revised" the state.

Research Program

We first instrument a CKM-style agent so that every claim $c$ in the topic state carries a support set. The support set lists the source papers it was extracted from and the parent claims it was derived from during contrastive updates. Together these form a directed dependency graph over claims and papers. When paper $r$ is retracted at time $t$, a claim keeps its status only if it retains support that does not pass through $r$:

$$\text{keep}(c) \iff \exists\, \text{a support path from some paper } p \neq r \text{ to } c \text{ that avoids } r.$$

Here a support path is a chain of extraction or derivation edges in the dependency graph. Claims with no surviving path are moved to the tracked do-not-synthesize set. Claims with weakened but surviving support are downgraded and re-verified against their remaining sources with a claim verifier in the style of SciFact. Hypotheses whose rationale cites removed claims are flagged as invalidated.

Errata are handled more narrowly. The correction notice is aligned to the specific claims extracted from the corrected element, such as a figure, table, or numerical result, and only those claims and their descendants are cascaded.

Following Öztürk et al., citing papers that critique the retracted work are treated as corrective rather than contaminated. Their "Criticism Exception" warns that uniform purging creates false positives. We therefore classify citation context before cascading through citing papers.

The second component is the two-tier state. New claims enter the provisional tier. They are promoted to the anchor tier only when an independent paper, one with no shared authors and not citing the first as its sole basis, supports them. Hypothesis generation may use provisional claims, but every hypothesis records which tier it depends on.

The hard part is extracting provenance faithfully from an LLM-driven update. When the model writes "evidence is strengthened," we must log which inputs it used. We will require structured update outputs that cite claim IDs, and we will audit a sample by hand.

Baselines

The naive reference point is unmodified CKM: it ingests everything and never acts on corrections. It establishes how much contamination accumulates and how much hypothesis quality degrades.

The second baseline is ingestion-time blacklisting. Every incoming paper is checked against retraction metadata of the kind compiled in the Retraction Watch-derived dataset of Öztürk et al., and already-retracted papers are excluded. This is the cheapest reasonable defense. It is blind to papers retracted after ingestion, which is precisely the case we study.

The third baseline is naive deletion: when a retraction arrives, remove the claims extracted directly from that paper and nothing else. This is the strongest cheap competitor. The ablation from it to our method isolates the key ingredient, which is cascading through derived claims.

The fourth baseline is LLM self-repair. The agent receives the retraction notice and its current topic file and is prompted to revise the file. This tests whether provenance bookkeeping is necessary at all.

The strongest method, and our oracle, is counterfactual replay. We re-run the agent from the last checkpoint before the retracted paper was ingested, with that paper removed, for the same number of windows. The replay state defines "what the agent would believe had it never seen the bad paper."

Beyond these baselines, we ablate the provisional tier to measure pre-retraction exposure, and we disable the do-not-synthesize set to test whether re-ingested restatements of retracted findings slip back in.

All systems use the same base LLM, the same streams, the same window schedule, and the same verifier. Token spend is logged per system. Prompt tuning gets an equal budget, fixed on held-out topics. No reported result was found on Lacuna for residual contamination, rollback fidelity, or retraction handling in any streaming literature agent. The reference numbers are therefore CKM's own: 72% topic coverage, 399-day median lead time, a 3.8% hit rate for convergence-triggered hypotheses, and a 91% token reduction relative to batch processing (CKM).

Beating the field means two things. First, residual contamination must fall well below naive deletion and self-repair, coming within a small margin of replay. Second, cost and churn of unrelated claims must be clearly below replay, and CKM-level hit rate and lead time must be preserved.

Why not just…?

Why not just re-run the stream without the retracted paper? This is the real threat. CKM costs roughly $3 per topic, so replaying a few windows may be cheap. We treat replay as the oracle, not a strawman. We expect our method to win on two axes rather than on quality. The first is cost under realistic correction rates: more than 10,000 retractions were issued in 2023 alone, and errata are far more numerous, so replays compound across topics and events. The second is churn. LLM pipelines are not deterministic, so a replay rewrites many claims unrelated to the retraction, and any downstream user or citing system sees spurious changes. We measure churn directly. If replay turns out cheap and low-churn in practice, we will report that rollback's value lies mainly in the provisional tier and in auditability, and narrow the claim accordingly.

Why not just filter retracted DOIs at ingestion? It cannot act during the window of risk, which by construction is the period before the notice exists. Over 30% of retraction notices also give no reason (Cheng et al.), so a binary filter cannot tell administrative retractions from data fabrication. The blacklist baseline will show near-zero benefit on post-ingestion retractions.

Why not just tell the LLM about the retraction and let it fix its notes? Evidence from AutoSupervision and RSI for Science suggests that models report fixes they did not make and that self-evaluation drifts optimistically. Self-repair is still a fair contender. Our evaluation measures the actual state against the replay oracle, not the model's description of its edits, so the experiment will settle the question.

Why not just wait a year before ingesting anything? Delay kills the lead time that is the point of streaming agents, and many retractions take longer than any reasonable delay. The provisional tier is the compromise. Comparing it against fixed ingestion delays of 6 and 12 months will show the trade-off between lead time and contamination.

Proposed Experiments

The first experiment uses controlled injection on the CKM 50-topic arXiv benchmark. In each topic stream we insert synthetic flawed papers, written to state a plausible but false result in the topic's vocabulary. Each is "retracted" 2, 4, or 8 windows later. Because the counterfactual is known exactly, we measure residual contamination as the fraction of claims in the post-correction state that are absent from the replay state and are entailed by the injected claim. Entailment is judged with a SciFact-style verifier and audited by hand. We pick the verifier carefully because open-domain verification is fragile: MultiVerS falls from 72.5 to 52.4 F1 on SciFact-Open. We therefore verify against the small local state rather than an open corpus. Alongside residual contamination we report churn of unrelated claims, token cost, maximum drawdown of hypothesis hit rate across windows, and CKM's lead-time and coverage metrics. The decisive check is whether rollback closes most of the gap between naive deletion and replay as the retraction delay grows. Residue should grow with delay for naive deletion and stay flat for rollback.

The second experiment uses real retractions. From the 3,193 content-related retractions and 7,424 citation pairs in the Öztürk et al. dataset, we select biomedical topics with dense retraction activity and build time-ordered streams from publication dates. We then run all systems with retraction notices arriving at their true dates. Here the replay oracle is still computable, and we additionally test the Criticism Exception. Cascading through critical citing papers should be wrong, and we measure how often each system purges corrective content.

The third experiment is a smaller study of errata. Using correction notices for figure and axis errors such as the Drosophila mitochondrial haplotype correction, we ask whether narrow cascading updates only the affected numerical claims.

The idea is falsified if naive deletion leaves little residue, meaning derived echoes are rare in practice, or if replay matches rollback on cost and churn. Either outcome would itself be a useful finding about how streaming literature agents actually propagate error.