Changing a Safety Classifier's Policy by Editing Its Memory

Safety classifiers decide whether a prompt or response is "safe" or "unsafe" under some written policy. The policy itself keeps moving. Regulations change, companies revise acceptable-use rules, and a new deployment needs an exemption (clinicians may ask about overdose thresholds) that the old classifier was trained to refuse. Today, every such change means relabeling data and retraining the classifier, or writing the new rule into the classifier's prompt and hoping it obeys. We propose a third route. The classifier's weights stay frozen. Next to them sits an editable external memory, a lookup table of labeled example inputs, and the final verdict blends the classifier's own judgment with the labels of the nearest stored examples. To change the policy, you write new entries into the table. Those entries come from a large language model (LLM) that turns the text of a new policy clause into pairs of nearly identical requests that fall on opposite sides of the new line.

In two sentences: we treat a guardrail's policy as data in an external store instead of as weights, and we populate that store for a new clause with LLM-written "minimal pairs" (one violating, one permitted, differing only in what matters). The question is whether this tracks policy edits as well as retraining, especially exemptions and reversals, without disturbing the rules that did not change. Prompting alone appears to handle those edits poorly, and retraining risks damaging the unchanged rules.

Core Claim

Our claim is that a frozen, policy-following guard combined with a counterfactual example memory can absorb four kinds of policy edit: adding a category, adding an exemption, reversing a decision, and deleting a rule. On the edited clause it should come within a small margin of parameter-efficient retraining on the same synthetic data. On unedited clauses its accuracy should be unchanged, which retraining does not guarantee. The update should take seconds and be exactly reversible. We further claim that the pairing is the active ingredient. Memory filled with the same number of unpaired synthetic examples, or keyed by an off-the-shelf sentence encoder, should fail on exemptions, because benign and harmful requests that share wording sit close together in ordinary embedding spaces.

Three lines of prior work motivate this claim. AIR-BENCH 2024 breaks 24 regulations and corporate policies into 314 risk categories. It finds that fixed models miss many regulated categories and do not reflect policy shifts without retraining, so it documents the drift problem but offers no mechanism for following it. AEGIS2.0 trains a guard with a "Topic Following" blend so that it can obey a policy written into its prompt at inference time, and reports zero-shot gains such as NSFW moving from 0.699 to 0.772. This is the prompt-only route we must beat, and the policy stays entirely in text. Improving RAG for LLM-based Mental Health Risk Detection through Contrastive Embeddings shows that off-the-shelf retrievers confuse "leaving for vacation" with "leaving this world," and that contrastive fine-tuning of the retriever fixes this on a fixed binary task. We reuse its contrastive objective but ask whether it transfers to clauses the retriever never saw.

Research Program

The model is a frozen guard $G$ that reads an input $x$ and the policy text $\pi$, and produces $p_G(\text{unsafe} \mid x, \pi)$. Beside it is a datastore $\mathcal{M}$ of pairs $(k_i, y_i)$. Each key $k_i$ is a learned embedding of a stored example. Each value $y_i$ is that example's label under the current policy. Following the interpolation used by Generalization through Memorization: Nearest Neighbor Language Models, which mixes a nearest-neighbor distribution with a language model's softmax to predict the next token, we compute

$$p(\text{unsafe} \mid x) = \lambda(x)\, p_{\mathcal{M}}(\text{unsafe} \mid x) + (1-\lambda(x))\, p_G(\text{unsafe} \mid x, \pi).$$

Here $p_{\mathcal{M}}$ is the distance-weighted vote of the $k$ nearest stored examples to the key of $x$. The mixing weight $\lambda(x)$ is a small function of the nearest-neighbor distance, so memory dominates only when a stored example is genuinely close. MemSFT motivates the decision to freeze $G$: it keeps a post-trained backbone frozen and routes domain knowledge through an external memory with a learned per-token mixing weight, and it avoids the 31-point collapse in general ability (83.6 to 51.83 on Qwen3-14B) that full fine-tuning caused. MemSFT studied biology, geoscience, and law. We apply the same separation to moderation, where the "knowledge" being separated out is the policy itself.

Keys are computed from $x$ alone, without the policy text, so that editing the policy changes only the values and the inserted entries, never the existing keys. A small projection head on top of $G$'s hidden state produces each key. We train this head once with a triplet loss, the method taken from the mental-health retrieval paper, on minimal pairs drawn from a fixed set of training clauses. After that it is never retrained. Whether a single generic head separates near-misses for clauses it never saw is the first open question we test.

Memory is populated by LLM-generated counterfactual minimal pairs, built in the style of the paired ambiguous/specific query generation in Zero-Shot and Efficient Clarification Need Prediction in Conversational Search. There, a student trained on such pairs beat direct LLM prompting by 1.61 to 1.85 F1 while running about 80 times faster. Given a clause, the generator first writes a request that violates it. It then writes a minimally edited twin that the clause permits, using a different role, an authorization, or a benign purpose with the same surface topic. It also writes adversarial rephrasings of the violating request using the mutation styles that SafeHarbor and AIR-BENCH apply: social engineering, dialect shifts, and appeals to authority. Generated knowledge can be wrong. Generated Knowledge Prompting for Commonsense Reasoning found that 17% of self-generated statements were non-factual and that these caused 86% of its errors. We therefore filter every pair with a category-specific judge prompt in the style of AIR-BENCH's autograder, which reached 0.86 kappa with humans, and we audit a human-labeled sample to estimate the label-noise rate that remains. Because entries are explicit, a bad entry found later can be deleted, which retraining cannot offer.

The testbed is a policy-edit protocol built from existing benchmarks. Our starting policy is the AEGIS2.0 taxonomy (12 categories, 9 subcategories) and the subset of SALAD-Bench's 66 categories that remains after holding some out. Edits come from three sources: held-out SALAD-Bench categories serve as "add category" edits, cases where AIR-BENCH's corporate policies disagree serve as "reverse" and "delete" edits, and clinical and administrative carve-outs serve as "add exemption" edits. For exemptions and reversals the file names no existing labeled near-miss benchmark, so we will collect a human-written, human-labeled evaluation set for each edit. Its authors will never see the generator's outputs, so the evaluation stays separate from the augmentation. To study memory scaling, we accumulate edits from many tenants' policies into one store, up to tens of thousands of entries, and measure whether retrieval precision and unchanged-clause accuracy degrade as the store grows. SafeHarbor's finding that flat rule retrieval returned about 78% irrelevant material warns that growth may hurt.

Baselines

The naive reference point is the frozen guard with the original policy prompt and no edit applied. Its errors on edited clauses set the floor, and its scores on unchanged clauses set the target that every method must preserve.

The prompt-only baseline is the AEGIS2.0 Topic Following guard, Llama3.1-AegisGuard, with the edited clause written into its system prompt. If that model is unavailable, we train a guard with the same recipe on the same base model we use. This is the strongest zero-retraining competitor because it was built for exactly this purpose. It runs on identical inputs with the same clause text our generator sees. We count the proposal as beating it if edited-clause F1 is higher on exemption and reversal edits by more than the seed-to-seed spread, and unchanged-clause F1 is no worse. We have not found a reported result on Lacuna for any guard on a policy-swap protocol of this kind. The closest reported figure is AEGIS2.0's 0.699 to 0.772 zero-shot gain on NSFW tasks.

The strongest retraining baseline is LoRA fine-tuning of the same guard on exactly the filtered synthetic pairs that go into memory, plus replay of the original training data to protect old clauses. It gets the same data and a matched tuning budget: we sweep rank and learning rate over as many configurations as we spend tuning $\lambda$ and $k$, and we report wall-clock update time. Beating it means landing within 2 F1 on the edited clause while showing less unchanged-clause degradation, or showing equal accuracy with exact reversibility and update times orders of magnitude shorter. As an absolute anchor for a strong fine-tuned judge, SALAD-Bench reports MD-Judge at F1 0.873 on its enhanced test set, against 0.827 for GPT-4 and 0.085 for LlamaGuard (SALAD-Bench). We will report our numbers on that same set for unchanged clauses.

We also run the closest existing fused system. RigorLLM combines a nearest-neighbor classifier over embeddings, trained on harmful examples it synthesizes, with a fine-tuned LLM guard. It reports a 23% relative F1 gain over the best baseline on ToxicChat at about 1.02 seconds per query. We feed it the same edit data in its kNN component, which tests whether our pairing and edit protocol add anything over RigorLLM's static fusion.

Two ablations isolate the key ingredient. The first fills memory with the same number of unpaired, filtered synthetic examples per clause. The second keys memory with an off-the-shelf sentence encoder instead of the contrastive head.

Why not just…?

Why not just write the new rule into the guard's prompt? This is the most serious threat, and it may well be sufficient for "add category" edits, where the new rule names a topic the model already understands. Our reason to doubt it on exemptions and reversals is indirect. SafeHarbor reports that retrieved-rule prompting is limited by how well the base model reasons and that weaker models fail. AgentHarm shows that a frontier model's refusal behavior swings sharply under a jailbreak template: Claude 3.5 Sonnet's refusal rate fell from 85.2% to 16.7% (AgentHarm). Neither result proves that prompt-only policies fail on carve-outs, so the edit-type breakdown in our main experiment decides it. If prompt-only matches memory on exemptions, we will report memory as unnecessary for that edit type.

Why not just LoRA-tune on the synthetic pairs? LoRA with replay may match memory on every accuracy metric. If it does, the remaining arguments are update cost, exact rollback, per-entry auditability, and behavior when many tenants' policies coexist, and we will say plainly that the accuracy claim failed. MemSFT's evidence that weight updates trade away general ability comes from domain knowledge, not moderation, so we measure interference directly instead of assuming it.

Why not just do retrieval-augmented few-shot prompting with an off-the-shelf retriever? Two results argue against it. SafeHarbor measured roughly 78% irrelevant retrievals for flat retrieval in agent guardrails. The mental-health study found that off-the-shelf embeddings retrieve opposite-label lookalikes, and that retrieved examples gave negligible or negative returns at 8B scale. We still run this as a baseline, with pairs placed in the prompt rather than interpolated, because it separates the value of the pairs from the value of interpolating labels.

Why not just use a plain nearest-neighbor classifier and drop the LLM? RigorLLM finds that nearest-neighbor votes resist gibberish suffixes but needed a generative guard alongside them to perform well on conversational data. A memory-only variant, with $\lambda = 1$, is part of our ablations. If it matches the blend, the guard is dead weight and the system gets much cheaper, which would itself be a useful finding.

Proposed Experiments

The decisive experiment is the policy-edit protocol, reported separately for each edit type. Metrics are edited-clause F1, unchanged-clause F1 on SALAD-Bench, AEGIS2.0, the OpenAI Moderation set, and ToxicChat, over-refusal rate on the human-written permitted twins, and update time. We expect memory to beat prompt-only on exemptions and reversals and to tie on added categories. We expect it to come within 2 F1 of LoRA-with-replay on the edited clause while showing no drop on unchanged clauses, compared with a measurable drop for LoRA without replay.

The second check is transfer of the contrastive key head. We train it on minimal pairs from training clauses only, then measure nearest-neighbor label precision on held-out clauses against an off-the-shelf encoder. If it fails to transfer, the proposal reduces to per-clause retriever training and loses its zero-retraining claim.

The third check probes robustness. We run SALAD-Bench's attack-enhanced subset and AdvBench jailbreak suffixes against every system after the edits. We also test an adaptive attacker who knows the memory exists and writes inputs designed to land near permitted twins, which is the obvious weak point of an exemption memory.

The fourth check tests sensitivity to generation quality and to memory scale. We inject 0%, 10%, and 20% label noise into generated pairs, and compare (a) leaving them in place with (b) deleting the entries the judge flags. We also grow the store across accumulated tenant policies and plot unchanged-clause accuracy against store size.

Together these show whether a guardrail's policy can live in data the deployer edits, or whether prompting or retraining already does the job. Each outcome tells practitioners which of the three routes to use for which kind of policy change.