Where Did the Update Go? Certifying Declared Change Envelopes for Model Updates

Muhammad Usama, Muhammad Saad QadirarXiv 2026 · arXiv

When we update a large language model (LLM)—say, to improve its medical knowledge or patch a specific policy—how do we know we haven't accidentally broken its performance on legal reasoning or basic math? In the world of software engineering, we’d use regression tests. But for LLMs, behavior is probabilistic, and "useful" updates are meant to change things.

This paper, "Where Did the Update Go?", introduces a formal framework called Declared Change Envelopes to solve this problem. It provides a way to statistically certify that a model update stays within its intended "footprint" without requiring expensive ground-truth labels for deployment traffic.

TL;DR

The authors propose "declared change envelopes" to certify that LLM updates only affect pre-specified regions of input space. By "anchoring" updates to the incumbent model's decisions on out-of-scope traffic, they demonstrate that narrow updates can achieve certifiably low leakage (under 5%) across various training methods and model sizes.

What Problem Does This Paper Address?

Model updates are notoriously "leaky." A narrow fine-tuning session on medical data can inadvertently shift the model's behavior across unrelated domains. Regulators (like those behind the EU AI Act) are beginning to require evidence that updates behave as declared.

Existing checks fall short for several reasons:

  • Regression testing requires labels, which are rarely available for live deployment traffic.
  • Global disagreement metrics are useless because a good update should change some decisions; we just want to know where those changes happen.
  • Model equality tests are too strict, rejecting any update that changes anything at all.

The authors fill this gap by defining leakage: the rate at which a model changes its decisions on inputs that were declared to be "out-of-scope" before training began.

Table 1: Comparison of this work with existing model update checks.

Methods

The paper introduces a three-part approach:

  1. Commitment Protocol: Before training, the developer commits to a "scope" (e.g., a rule-based filter or a natural language description evaluated by a judge). This prevents "drawing the target around the arrow" after seeing the update's effects.
  2. Anytime-Valid Certification: The authors use Beta-mixture martingales to create confidence sequences. This allows an operator to continuously monitor live, unlabeled traffic and stop the audit as soon as they have enough evidence to certify the update, without the statistical "peeking" issues of fixed-sample tests.
  3. Decision Anchoring: To make updates certifiable, the authors add a consistency term to the training objective. This term penalizes the KL divergence between the new model and the incumbent model's output distributions on unlabeled prompts outside the declared scope.

Datasets

The study evaluates updates on a massive pool of 34,136 queries across 11 domains:

  • MMLU & MedMCQA: Medical, legal, STEM, and humanities.
  • Reasoning & Logic: ARC, CommonsenseQA, and GSM8K (math).
  • Classification: AG News (topic), SST-2 (sentiment), and CaseHOLD (legal).
  • Knowledge Editing: CounterFact.
  • Open-ended Chat: OpenAssistant.
  • Safety: XSTest and AdvBench.

The models tested range from 1.5B to 72B parameters, including the Qwen2.5 and Llama-3.1 families.

Key Insights and Ideas

  • Standard updates are incredibly leaky: Plain LoRA or full fine-tuning updates often keep only 7–36% of their changes inside the declared scope. A medical update might change 10% of the model's decisions on sentiment analysis.
  • Leakage tracks KL divergence: There is a near-perfect correlation (Spearman 0.94) between decision leakage and the update’s KL divergence to the incumbent on out-of-scope inputs. This suggests that controlling the output distribution is the direct lever for controlling behavior drift.
  • The "Self-Disagreement Floor": LLM inference is not bitwise reproducible. Simply re-batching the same model can change ~1% of multiple-choice decisions and up to 21% of open-ended chat responses. This "noise floor" sets a fundamental limit on how low a leakage threshold can be certified.
  • Anchoring is the "Aha!" moment: Traditional regularizers like L2 penalties or weight interpolation fail to reach certifiable levels without destroying the intended update. Anchoring specifically to the decision format (e.g., multiple-choice options) on unlabeled data is what makes narrow updates certifiable.

Key Results

  • Certifiable Leakage: Anchored updates (LoRA, GRPO, and full fine-tuning) can certify a 5% leakage bound using only a few hundred to a few thousand unlabeled samples.
  • Composition: Certificates compose linearly. If you have four updates each certified for 5% leakage, you can bound the total lifetime drift of the system.
  • Preserving Gains: Anchoring generally preserves the intended performance gains (e.g., +80 points on a policy change) while slashing unintended leakage by 3x to 7x.
  • Scale Invariance: The effectiveness of anchoring holds steady from small 1.5B models up to 72B giants.

Limitations and Future Work

  • Decision-like outputs only: The framework works beautifully for multiple-choice and short-form answers. However, for open-ended chat, the numerical "noise floor" is so high that no update can currently be certified for low leakage.
  • Strata Masking: A global 5% leakage certificate might hide the fact that one specific sub-domain (a "stratum") was 100% rewritten. The authors suggest that critical sub-domains should be declared and certified separately.
  • Label-Free Limitations: The certificate tells you where the model changed, not if the changes were good. It is a measure of locality, not quality, and must be complemented by traditional accuracy benchmarks.
  • Future Directions: Certifying open-ended generation remains an open challenge, likely requiring batch-invariant inference techniques or more sophisticated semantic judges.

Summary updated

Abstract

Deployed language models are updated continually, and regulators increasingly accept changes that were declared in advance. Showing that an update changed behavior only where it was declared to is a different statistical problem from bounding how much it changed: a useful update is meant to change some decisions, so global disagreement and regression bounds cannot certify it. We formalize the problem with declared change envelopes. Before training, the developer commits to a region of inputs, and every decision that changes outside it counts as leakage. Leakage compares two models, so it can be certified from unlabeled traffic with anytime-valid tests whose sample cost we bound from above and below, and per-update certificates compose into a bound on lifetime drift outside all declared scopes. We measure leakage for LoRA and full fine-tuning, preference optimization, reinforcement learning, and knowledge editing on models from 1.5B to 72B parameters. Narrow updates leak widely: domain and policy LoRA updates keep only about 7-36% of the decisions they change inside the declared scope. Across 256 trained variants, leakage tracks the update's KL divergence to the incumbent on out-of-scope inputs (Spearman 0.94). None of the customary remedies we tested makes a rule-like update certifiable without giving up much of its intended change. Distilling the incumbent's decisions on unlabeled prompts in the deployment format does: anchored LoRA, GRPO, and (at a small learning rate) full fine-tuning updates certify 5% leakage from a few hundred to a few thousand unlabeled samples, and the certificates replicate on held-out traffic and on written answers. The certificates cover decision-like outputs only; on open-ended chat, re-batching alone changes 11-21% of responses and no update certifies. Code: https://github.com/Usama1002/declared-change-envelopes
BibTeX
@misc{usama2026where,
  author = {Muhammad Usama and Muhammad Saad Qadir},
  title = {{Where Did the Update Go? Certifying Declared Change Envelopes for Model Updates}},
  year = {2026},
  eprint = {2611.04192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04192},
}
Figure