Whether a chatbot should refuse, warn, or simply help often depends on where the user is and when they ask. Laws on data protection, reproductive care, narcotics, or speech differ by country and get amended. Current systems for jurisdiction-aware safety train each country's rules into the model's weights, so every amendment means another round of training. The obvious fix is to look up the current rule at answer time with retrieval-augmented generation (RAG), where the system searches an external document store and pastes relevant passages into the prompt. Prior work shows this naive fix fails in two opposite ways. Pasting retrieved text into the prompt makes safety-trained models more permissive, and pasting sensitive statutory language makes them refuse harmless questions. On top of both, models tend to follow the old version of a fact even when the new one is in front of them.
This note proposes that the problem splits into two separate questions, each with its own fix. The first is which rule is currently in force for this user. We argue that question should be answered outside the model, by a versioned rule store that tracks which rule amends or replaces which and resolves the active rule set deterministically. The second is whether the model obeys that rule over its training-time prior and over the pull of "just complete the task." We argue that question should be handled by delivering resolved rules through a separate, trained constraint channel instead of mixing them with retrieved evidence. We build a benchmark of rule amendments and test the two fixes separately and together in a factorial design.
Core Claim
Retrieval can keep localized safety current without degrading it, under two conditions. First, it must retrieve resolved, versioned rules rather than raw statute text. Second, the model must be trained to treat a dedicated rule block as binding, including when the rule contradicts what it learned in pretraining.
The claim is falsifiable. Take a model that is given the correct current rule as ordinary retrieved context. Our prediction is that it will still show a measurable rate of stale-rule answers, the same pattern of sliding back to the old rule that StateMemBench documents. It will also show the refusal shifts reported for retrieval-equipped agents. Moving the same text into a trained constraint channel should remove most of both effects, and external resolution should remove most of the rest. If the plain-context model already follows amendments reliably, the core claim is wrong.
Where This Sits
Three papers motivate the gap by documenting failures.
Information Retrieval Induced Safety Degradation in AI Agents shows that giving aligned models Wikipedia or web retrieval pushes them into a "task-execution mode." In that paper, LLaMA 3.2's refusal rate on biased prompts dropped from 0.80 to 0.46 with web access, and safe system prompts did not stop the drop. That work diagnoses the problem on general English harms with a fixed retrieval snapshot. We study whether the placement of retrieved normative text, rather than its presence, causes the degradation.
Steering Over-refusals Towards Safety in Retrieval Augmented Generation shows the opposite failure: hazard words in retrieved passages trigger refusals of benign queries. It fixes this with a fixed steering vector added to the model's middle layers. A single fixed vector cannot tell a refusal a local statute requires apart from one caused only by statute vocabulary, and that distinction is exactly what jurisdiction-aware compliance needs.
Can Agent Memory Systems Track Evolving State? shows that models anchor on older, more prominent facts even when both old and new facts are in context. It fixes this with StateMem, which stores facts as units linked by dependency edges and uses a deterministic rechecker to decide what is current. We reuse that idea directly for our resolver. StateMem was tested on shopping, research, and finance scenarios, not on legal amendment, repeal, precedence between national and regional law, or grandfather clauses.
Two further papers supply parts of the method, and both constrain it.
Controllable Safety Alignment (CoSAlign) trains models to follow natural-language safety configurations placed in the system prompt. This is our constraint channel. CoSAlign assumes a human writes the correct, current configuration. We fill the channel automatically from a resolver and train specifically on rules that contradict the model's priors.
Safety Compliance trains a Compliance Reasoner with supervised fine-tuning and GRPO, a reinforcement-learning method that compares groups of sampled answers. Its training data are EU AI Act and GDPR scenarios built from hierarchical "Law Trees." We borrow its Law Trees as the backbone of our rule store and its scenarios as a starting point for our evaluation data. Its knowledge lives in the weights, so amendments are exactly its weak point.
SafeWorld contributes GEOSAFEDB, a database of norms and public policies for 50 countries. It also serves as the "retrain on every change" reference: its model is aligned with DPO (direct preference optimization) on a 2024 snapshot.
One paper cuts against our proposal, and we address it head on. SEA-SafeguardBench reports that telling a model to "follow local norms" fails when the model saw little regional data in pretraining. Our bet is narrower. An explicit, resolved rule ("in jurisdiction J after date D, X is prohibited") asks less of the model's regional knowledge than a vague instruction does. We do not assume the bet holds. We report results separately for high- and low-resource regions.
Research Program
The rule store. We build a versioned rule store. Each rule unit records its jurisdiction, subject scope, effective and expiry dates, and typed links to other rules: amends, repeals, overrides, and exception-to.
Legal content comes from the Safety Compliance Law Trees for the EU AI Act and GDPR and from the policy portion of GEOSAFEDB. Given a user's jurisdiction and a query date, a deterministic resolver walks these links and returns the rules in force at that date. When the store has no rule matching the jurisdiction and topic, the resolver returns an explicit "no local rule found" marker. This marker targets a known failure: the model silently falling back to its Western-default assumptions, a pattern RealTime QA documents for stale facts.
The benchmark. We build an amendment benchmark, called AmendSafe here. Each item is a user query whose correct handling flips across a rule change: before the change the right response is to help, after it the right response is to warn or refuse, or the reverse.
Items come in two kinds. Real items use actual staged or amended provisions. Counterfactual items use synthetic amendments, generated with CAGE-style placeholder templates (CAGE) that swap in local entities. Because no model has seen a counterfactual amendment in pretraining, these items separate "knows the new rule from pretraining" from "follows the retrieved rule."
Each item has a benign twin: a harmless query whose retrieved rule text still contains hazard vocabulary. The twins measure over-refusal in the way RAGRefuse does.
The model. We fine-tune Llama-3.1-8B and Qwen-8B in the CoSAlign style. Training episodes deliberately rotate the rule block across dates and jurisdictions, and a substantial share of blocks contradict the model's likely prior. This teaches the model to obey the block even when its weights disagree. Retrieved factual passages go in a separate evidence block that the model is trained to treat as information, not instruction.
The factorial study. The analysis crosses two factors. The first is who decides which rule is current: the model, given dated raw statutes, or the external resolver. The second is where rules appear: mixed in with evidence, or in the trained constraint channel. The four cells show how much of the failure comes from resolution and how much from delivery.
Baselines
The naive reference is the untuned base model with ordinary RAG over the raw, dated statute corpus, with passages ordered by a cross-encoder reranker. This is what a practitioner would build first. The retrieval-degradation paper predicts both lower refusal of harmful requests and stale answers for this setup. Any proposed system must beat it on both current-rule compliance and benign-twin over-refusal.
The parametric reference is a Compliance-Reasoner-style model: supervised fine-tuning plus GRPO on the pre-amendment Law Tree scenarios, following Safety Compliance, with no retrieval. That paper reports 77.27% average accuracy for Compliance-Reasoner-GRPO (Qwen-8B) on its EU AI Act benchmark. We first reproduce that number on unamended items, then measure the drop on post-amendment items. We also report a "retrained" variant that is re-fine-tuned on post-amendment data. That variant gives an upper bound together with its cost in annotation and GPU-hours. Our system beats this baseline if it matches the retrained variant's post-amendment accuracy within a few points at negligible update cost, and clearly exceeds the non-retrained model.
The strongest inference-time competitor is CoSAlign given the same rule text our resolver returns, but written as a single system-prompt block without typed version links and without counter-prior training. Controllable Safety Alignment reports a CoSA-Score of 0.408 for CoSAlign against 0.217 for in-context alignment on Llama 3.1 8B. We rerun CoSAlign's released recipe with the same base model, the same training-token budget as our fine-tune, and the same hyperparameter search. No reported result was found on Lacuna for CoSAlign, or any method, on rule changes over time. Beating this baseline therefore means a statistically clear gain on counterfactual post-amendment items, where parametric priors are most misleading.
The over-refusal competitor is SafeRAG-Steering applied to the naive RAG pipeline. On RAGRefuse, Steering Over-refusals reports that it cut Llama-3.1-8B over-refusal from 53.4% to 4.3%. We run it on our benign twins and on the legally required refusals. The prediction is that it lowers over-refusal but also suppresses legitimate refusals that only a specific local rule mandates. We beat it if we match its benign-twin over-refusal while keeping required-refusal recall substantially higher.
Two ablations isolate the key ingredients. Removing counter-prior training while keeping the channel and resolver tests whether rotating contradictory rules is what overcomes the prior. Removing the "no local rule found" marker tests how much abstention matters when retrieval comes up empty.
Why not just…?
Why not just put the current rules in the system prompt? This is the most serious threat, and the factorial study is built to settle it. Plain in-context alignment trails trained configuration-following by a wide margin: 0.217 against 0.408 CoSA-Score in Controllable Safety Alignment. Safe prompting also failed to stop retrieval-induced degradation in Information Retrieval Induced Safety Degradation. The harder version of the objection is "CoSAlign fed by a resolver is enough." If that is what we find, we will report it plainly. The contribution then shrinks to the resolver and the amendment benchmark, and the counter-prior training claim is dropped.
Why not just re-run DPO or GRPO whenever a law changes? This works between updates. It costs new preference data and a training run per jurisdiction per amendment, and the model stays wrong during the lag. With 50 countries in GEOSAFEDB and continuous amendment, that lag is the normal state, not an edge case. The retrained parametric baseline measures this trade-off directly: accuracy that matches ours at orders of magnitude higher update cost would confirm the argument. Clearly higher accuracy would weaken it.
Why not just retrieve dated statutes and let a strong reasoning model work out which is current? StateMemBench shows that models anchor on stale facts even under oracle retrieval, and that chain-of-thought reasoning can make this worse. StateMem's structured resolution raised current-state accuracy from 0.205 to 0.363 (Can Agent Memory Systems Track Evolving State?). Legal precedence is harder than a budget update, so we expect the gap to widen. The model-resolves cell of the factorial study tests exactly this, including with reasoning traces enabled.
Why not just steer away over-refusals? A fixed steering direction cannot depend on which rule is in force. On benign twins versus mandated refusals, it should lower both kinds of refusal together, which is the failure the SafeRAG-Steering baseline comparison is designed to expose.
Proposed Experiments
The first decisive check is the counterfactual-amendment test. On synthetic rule changes that no pretraining corpus contains, we measure the stale-rule rate: the fraction of answers that follow the superseded rule. We report it for all four factorial cells. The core claim requires two things. Raw context should show a substantial stale-rule rate even when the model resolves versions correctly. The constraint channel plus resolver should bring that rate near zero.
The second check tests whether the channel protects against retrieval-induced degradation. We attach the same pipelines to XSTest_v2, SafeArena, and BBQ with rule and evidence retrieval switched on, and compare refusal of harmful requests against the no-retrieval model. The prediction is that the evidence-mixed cell reproduces the drop reported in Information Retrieval Induced Safety Degradation, and the channel cell does not.
The third check measures the balance between over-refusal and required refusal on AmendSafe benign twins, plus the RAGRefuse legal domain as an outside reference. We plot benign over-refusal against recall on mandated refusals for our system, SafeRAG-Steering, and naive RAG.
The fourth check covers empty retrieval and uneven regional data. For held-out jurisdictions we delete the relevant rules from the store and measure how often the model hedges or asks, versus silently applying a Western default. We report all metrics separately for high- and low-resource regions, following the concern raised by SEA-SafeguardBench.
Finally, we check that general ability is preserved on MMLU and HellaSwag, using the same no-degradation standard SafeWorld reports for its model.
References
- Information Retrieval Induced Safety Degradation in AI Agents
- Steering Over-refusals Towards Safety in Retrieval Augmented Generation
- Can Agent Memory Systems Track Evolving State?
- Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements
- Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
- SafeWorld: Geo-Diverse Safety Alignment
- SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures
- RealTime QA: What's the Answer Right Now?
- CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation