Language models increasingly have to track facts that change while they read: a character moves rooms, a price changes, a policy is revised. There are two broad ways to keep up. The model can hold recent text in its context or retrieve it from a store. Or it can update its own weights on the incoming text, an old idea called dynamic evaluation, meaning small gradient steps taken during inference. These families have never been compared on equal terms.
The benchmark built for fact-tracking tests only the first family. The work that revived weight updating measures only perplexity on old books. Neither lets the incoming text keep arriving while the model computes. In image classification, making the stream keep moving during adaptation reversed which methods look best. We propose to build the same kind of clock-respecting protocol for language models tracking streaming facts. We will use it to test whether weight updating actually helps when every method pays for its own compute.
Core Claim
Our central hypothesis is that the ranking of fact-tracking strategies depends on the evaluation protocol. Under the usual "paused-world" setup, where the stream waits for the model, heavy methods look good. Once text arrives at a fixed rate and a method that falls behind must answer with stale state, the ranking changes.
We further hypothesize that weight updating and context-based methods fail in different ways:
- Weight updating should resist being pulled off a correct answer by irrelevant intervening text. It should, however, register new facts more slowly, because a single gradient step moves the weights only a little.
- Retrieval should register new facts quickly. It should also frequently fetch an outdated version of a fact.
The method we contribute is one cheap ingredient borrowed from vision. Under time budgets, the image method that skipped uninformative samples did best. We therefore take gradient steps only on text chunks where the model is surprised, meaning its next-token loss is high. We reset the low-rank adapter when high surprise persists. The claim is testable because gating on surprise must beat gating at random at an identical update count.
Relation to Prior Work
Two papers motivate the gap directly.
- Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams introduces OAKS. It evaluates 14 models every 2k tokens with three diagnostic metrics:
- acquisition latency: how many chunks pass before a changed fact is picked up;
- distraction susceptibility: how often a correct answer flips to wrong because of irrelevant text;
- phase miss rate: the fraction of time intervals whose valid fact is missed entirely.
It explicitly leaves weight-updating adaptation unbenchmarked. It also does not charge methods for wall-clock time.
- Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models supplies the adaptation machinery we reuse: online LoRA on middle layers, n-step update frequencies, and weight resets. It reports that a 150M dynamic model can match a 1B static model under shift. Its evaluation, however, is log-loss against FLOPs on PG-19 alone, and its reset points are chosen by hand.
We differ from both. We run the second paper's methods on the first paper's fact-tracking tasks, under a time-coupled protocol.
Two further papers supply and constrain the protocol.
- Evaluation of Test-Time Adaptation Under Computational Time Constraints defines the relative-speed quantity $\mathcal{C}(g)$: a method $\mathcal{C}(g)$ times slower than the stream adapts only on every $\mathcal{C}(g)$-th batch and predicts the rest with its latest weights. Under this protocol, simple or sample-rejecting methods overtook newer ones on ImageNet-C. We port this protocol from images to token streams. We also extend it from constant arrival rates to bursty ones, which that paper leaves open.
- UniTTA shows that correlated, non-i.i.d. streams change conclusions. It ignores compute time. We borrow its lesson by varying how quickly facts change.
Two papers constrain the design.
- Parameter-free Online Test-time Adaptation documents "silent collapse," where weight updates on correlated streams drive the training loss down while accuracy falls. This is why resets and a frozen-weight control are mandatory in our study.
- The Design and Implementation of a Scalable Deep Learning Benchmarking Platform measures up to 64% CPU slowdown from Python-level overhead. This is why we report both FLOPs and wall-clock time. It is also why we test whether the two accountings agree.
Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline is the cautionary precedent. It lifted PointNet++ from 90.0% to 93.3% on ModelNet40 purely by fixing the training protocol and exposed test-set peak picking. Accordingly, we give every method an identical tuning budget and select hyperparameters only on held-out streams.
Research Program
We will build a streaming harness around OAKS-B (synthetic, BABILong-derived, 1.2k questions, about 4.7 changes per question) and OAKS-N (39 novels, 870 questions).
How the stream runs. Text chunks arrive at a controlled rate $r$ in tokens per second of wall-clock time on fixed hardware. We use two arrival schedules: constant, and bursty Poisson-style arrivals. Questions are posed at fixed points in the stream. A method answers a question with whatever state it has at that moment, whether that is a context window, a retrieval index, a memory, or adapter weights. Any chunk that arrives while the method is busy is either queued, if the method's design allows it, or skipped for adaptation. We run two schemes, following Alfarra et al.:
- dual-pipeline, where the model answers using its latest weights while an update runs in the background;
- single-model, where no answer can be given while an update is in progress.
Sweeping the rate. We sweep $r$ from effectively infinite time per chunk (the paused-world protocol, which reproduces the original OAKS setup) down to rates at which even plain forward passes fall behind.
What each method is charged. Every method is charged its real latency. We log FLOPs alongside, so the two accountings can be compared.
The proposed method. The contribution on the method side is surprise-gated online LoRA. Let $\ell_t$ be the mean next-token loss on chunk $t$ under the current weights. We compute $\ell_t$ during the forward pass the model must run anyway, so it costs nothing extra.
- Update rule. We take a gradient step only when $\ell_t$ exceeds a running quantile of recent losses. The quantile is set so that the expected update rate fits the time budget.
- Reset rule. We restore the adapter to zero when $\ell_t$ stays above a higher threshold for $k$ consecutive chunks. This is our proxy for "the world changed beyond what the adapter tracks."
The reset rule addresses the hand-set resets in Krause et al. It is deliberately simple, so that the protocol question and the method question stay separable.
Models. We use open-weight models at three sizes spanning roughly 150M to several billion parameters. This lets us check whether the 150M-dynamic-versus-1B-static result on PG-19 carries over to fact tracking once compute is charged in time. Because OAKS-N risks pre-training memorization, OAKS-B is our primary testbed and OAKS-N serves as a naturalistic check.
Cost control. To contain the roughly 78k-inference cost reported for OAKS-B, we evaluate every method on a fixed, stratified subsample of questions for the full rate sweep. The complete set is used only at two rates.
Baselines
Paused-world reference. The naive reference is each model with no adaptation beyond its native context window, evaluated under the original OAKS protocol. For OAKS-B, OAKS reports 33.0% average accuracy across models and 66.3% for Gemini 3 Pro. For OAKS-N, it reports 57.5% average and 75.5% for Gemini 3 Pro. Gemini's weights are closed, so it anchors the paused-world scale but cannot enter the weight-updating comparison. The file gives no per-model OAKS numbers for open models, so we will re-establish them ourselves at the paused-world end of the sweep. Matching those averages in aggregate is our sanity check that the harness reproduces OAKS.
Rolling context window. Each model keeps the most recent tokens that fit within the same latency as the adaptive methods. This is the cheapest strong competitor, and it tests whether weights are needed at all.
Retrieval over chunks, plain and recency-weighted. OAKS found that semantic retrieval fetches stale chunks when consecutive chunks look alike. We therefore add a recency-weighted retriever, which is the obvious fix for that failure. It is plausibly the strongest non-weight competitor. No reported result was found on Lacuna for the accuracy of retrieval-based methods on OAKS, so we will measure it ourselves.
MemAgent. This agentic memory system is included as OAKS evaluated it. OAKS reports that it often trails plain retrieval and lags high-velocity streams. The file gives no reported numeric result for MemAgent on OAKS.
Dense dynamic evaluation. This is the strongest weight-updating competitor: online LoRA with a fixed n-step update frequency and session resets, exactly as in Revisiting Dynamic Evaluation. On PG-19 it reduced cumulative log-loss from 2.26 nats/token for the static model to 2.20. We tune its update frequency so that it uses the same wall-clock budget as our gated method.
Hybrid. We also test retrieval plus dense dynamic evaluation, to check whether the two failure modes complement each other.
Ablations isolating the key ingredient. Three ablations separate the parts of our method:
- random gating at the same number of updates, which isolates the surprise signal;
- gating without resets, which isolates the reset rule;
- resets on a fixed schedule, which checks whether surprise-based resets add anything over timing alone.
Fairness. All methods receive the same streams, the same hardware, and the same number of hyperparameter configurations tried on held-out validation streams. They are scored on the same questions. Beating dense dynamic evaluation requires two things:
- Higher accuracy on OAKS-B at matched wall-clock budget, at rates where dense updating falls behind.
- No loss at the paused-world end.
Beating retrieval requires lower distraction susceptibility with acquisition latency no worse than one extra chunk.
Why not just…?
Why not just compare methods at matched FLOPs, as dynamic evaluation already does? FLOPs ignore the fact that a backward pass cannot overlap with the forward pass that must answer the next question. They also ignore interpreter and memory-transfer overheads, which the MLModelScope study shows can be large. We report both accountings. If method rankings under FLOP-matching and wall-clock-matching agree across the sweep, that is itself a useful finding: it would mean the cheaper FLOP protocol suffices. We would state it as such rather than insist on the clock.
Why not just use a longer context window or a rolling buffer? This is a real threat on OAKS-B, where stories are short enough for long-context models. That is why the rolling window is charged the same latency and run at every rate. The PG-19 "context wall" result suggests weights win once relevant history exceeds the affordable window. On OAKS, however, this is unproven, and the rate sweep will show at which point, if any, it happens.
Why not just fix retrieval by weighting recency? This may close most of the stale-fact gap that OAKS attributes to retrieval, and we include it as a baseline for that reason. Our hypothesis predicts that recency-weighted retrieval will still show high distraction susceptibility on OAKS-N, where irrelevant but recent text is common. If it matches weight updating on all three diagnostic metrics at every rate, we will conclude that weight updating is unnecessary for this task family.
Why not just freeze weights and correct outputs, LAME-style, to avoid collapse? LAME's own authors note that output correction cannot fix wrong internal features. A new fact is exactly such a case: no reweighting of answer probabilities encodes that a character moved. We nonetheless run a frozen-weight control to detect silent collapse. This is the setting in which the adapter's training loss falls while accuracy on OAKS questions drops. If we observe it, it counts against weight updating in the results.
Decisive Checks
Does the protocol change the ranking? We compute Kendall's rank correlation between method orderings on OAKS-B accuracy at the paused-world setting and at each finite rate. A correlation well below one at practical rates confirms that paused-world benchmarking misranks these methods. A correlation near one at all rates refutes the core claim.
Do the two families fail differently? At matched budget, dense dynamic evaluation should show lower distraction susceptibility and higher acquisition latency than recency-weighted retrieval. The opposite pattern, or no difference, falsifies the hypothesis that the families fail in different ways.
Does the surprise signal matter? Surprise-gated updates must beat random gating at an identical update count, on both accuracy and phase miss rate, across seeds. If they do not, the gating ingredient carries no information and the method contribution fails. The protocol contribution would still stand.
Does the small-dynamic result transfer? We test whether the smallest dynamic model matches the largest static model with a rolling window at equal wall-clock time on OAKS-B. This checks whether the PG-19 result transfers from perplexity to fact tracking.
Is the effect real on OAKS-N? We compare gains on OAKS-N against OAKS-B. Gains that appear only on novels would point to memorization rather than adaptation.
References
- Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
- Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models
- Evaluation of Test-Time Adaptation Under Computational Time Constraints
- UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation
- Parameter-free Online Test-time Adaptation
- The Design and Implementation of a Scalable Deep Learning Benchmarking Platform
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline