Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates

Zehao Zhao, Shangjian Yin, Rui Liu, Chongyang Bai, Mark Ibrahim, Kavosh Asadi, Yuchen Lu, Hang Cui, Hamed FiroozarXiv 2026 · arXiv

This paper from Meta AI introduces Control Map, a new evaluation framework for test-time model updates. As Large Language Models (LLMs) increasingly use techniques like model editing or test-time training to adapt during inference, the authors argue that our current ways of measuring their success are fundamentally flawed.

TL;DR

Current evaluations of test-time model updates rely on "update counts," which hide critical differences in the timing and order of information available to a model. The Control Map framework fixes this by recording exact update histories and separately measuring immediate learning, long-term retention, and "collateral damage" to unrelated knowledge.

What Problem Does This Paper Address?

LLMs can now update their internal state (weights, adapters, or memory) while answering queries. Researchers usually compare these systems by looking at the "total number of updates" or "update frequency."

The authors point out a major blind spot: aliasing. Two systems might both be labeled as having "two updates," but one might answer a question after the first update, while the other answers after the second. If the second update contained the key information for the query, the first system is unfairly penalized. These coarse summaries omit what was updated, where in the model it happened, and when it occurred relative to the query.

Methods

The Control Map is an evaluation framework (not a new update algorithm) that tracks the ordered sequence of committed updates preceding every single model response. It evaluates performance across four non-compensating dimensions:

  1. Acquisition: Did the model learn the new information immediately?
  2. Retention: Does it still remember that information after more updates occur?
  3. Locality: Did the update preserve the model's performance on unrelated tasks?
  4. Transportability: Does the conclusion hold up in a different model or benchmark?

The authors also provide a mathematical "history-resolution certificate" (Theorem 1). This proves that if two systems have identical update histories, they must yield identical average scores under certain assumptions, allowing researchers to verify their experimental design before even looking at the results.

The Control Map tracks the exact history of updates (Z) preceding a score (Y), revealing differences that simple "update counts" hide.

Datasets

The study uses an extensive "history audit" across 27,072 rows from eight major evaluation sets:

  • General Reasoning: SocialIQA, BoolQ, PIQA, WinoGrande, HellaSwag, and ARC (Easy & Challenge).
  • Language Modeling: LAMBADA.
  • Model Editing: CounterFact, zsRE, WikiData-CounterFact, and WikiRecent.

These were tested across various model families, including GPT-2, Llama-3.2 (1B and 3B), Pythia-410M, and Qwen2.5-0.5B.

Key Insights and Ideas

  • Scheduling is a First-Class Citizen: The specific timing and order of updates are primary drivers of performance. Update scheduling alone can be the difference between a model appearing "smart" or "forgetful."
  • Learning $\neq$ Success: "Acquisition" (immediate learning) is a poor proxy for overall quality. A model can learn a fact perfectly but fail to retain it or cause massive "collateral damage" to its general knowledge.
  • No "Silver Bullet" Editor: Rankings of popular model editors (like ROME or MEMIT) are highly unstable. An editor that is "best" on one benchmark often fails miserably on another.
  • Input Length Matters: Longer inputs provide more "windows" for updates to occur, which progressively reveals the differences between nominally similar update schedules.

Key Results

  • Massive Performance Swings: In experiments with a 760M-parameter model, 24 different update schedules—all with the exact same total update count—yielded accuracies ranging from 19.7% to 34.2%.
  • Locality Failures: While methods like ROME and LoRA often pass acquisition tests, they frequently fail locality. In CounterFact tests, ROME retained edits well but failed locality at every single one of the 13 operating points tested.
  • Ranking Reversals: MEMIT’s locality advantage over ROME on the CounterFact benchmark completely reversed when tested on the zsRE and WikiData-CounterFact datasets.
Accuracy varies significantly across different update schedules, even when the total update count is fixed.

Limitations and Future Work

The framework's guarantees rely on the accuracy of the "tracing rule" used to identify relevant updates. If an update path is omitted (e.g., an unrecorded state change), the mathematical bounds may not hold.

The authors suggest that future work should focus on more automated tracing methods and that researchers should move away from reporting single summary scores. Instead, they advocate for providing full "evidence records" that include update histories, costs, and separate metrics for acquisition, retention, and locality.

Summary updated

Abstract

Test-time updating lets a language model change persistent state after training---such as model weights, adapter parameters, or a memory module---before or while answering queries. Existing evaluations often compare systems using update count or frequency and performance measured immediately afterward. These summaries omit what is updated, where, when, and in what order. The same total can therefore hide an important difference: one system may answer after one update while another answers after two. We introduce the Control Map, an evaluation framework that records the committed updates preceding each scored answer and separately evaluates behavioral changes, immediate learning (acquisition), later recall (retention), preservation of unrelated behavior (locality), and performance in new settings. We prove when counts alone determine final state and show, under stated assumptions, how differences between pre-answer histories limit differences in average scores. Using ControlMap, our experiments reveal three insights. First, update scheduling matters: at a 760M-parameter checkpoint, 24 schedules with the same update count yield 19.7%--34.2% accuracy. Second, learning an edit does not guarantee that it lasts or remains contained: all six selected CounterFact LoRA settings learn the new information, while ROME retains edits at all 13 operating points but fails locality at each. Third, no editor performs best consistently across models, benchmarks, and tests. Evaluations should therefore report update histories and assess acquisition, retention, and locality separately.
BibTeX
@misc{zhao2026beyond,
  author = {Zehao Zhao and Shangjian Yin and Rui Liu and Chongyang Bai and Mark Ibrahim and Kavosh Asadi and Yuchen Lu and Hang Cui and Hamed Firooz},
  title = {{Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates}},
  year = {2026},
  eprint = {2610.36109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36109},
}
Figure