Safety Checks as Code: Distilling Context-Conditioned Guards into Small Code Agents

Agents that act by writing Python, rather than emitting one tool call at a time, are faster and more capable. They are also dangerous in a context-dependent way. The same script that exports patient records is routine for a clinician and a violation for a receptionist. The same meal-planning routine is helpful for most users and harmful for a diabetic one. Current safety methods for language models judge such cases in text: they reason about the user, then decide. A code agent, however, often cannot know whether an action is harmful until the program has run. The query result may turn out to contain a minor's record, or the computed recipe may exceed a sugar limit. We propose teaching small code agents to write their safety reasoning into the program itself. Before each consequential operation, the agent writes an explicit guard, an ordinary Python check over the user's context and the values actually computed at runtime. The interpreter then refuses to perform consequential operations that no guard covers.

The bet is specific. Text safety reasoning has to predict what code will do. Small models are bad at that prediction, because they routinely hallucinate intermediate program state. A guard evaluated at execution time does not need to predict anything. So if we distill guard-writing from a large teacher into 0.5B–7B students, they should beat students distilled with text safety rationales, and the advantage should appear mainly on tasks where the harm depends on runtime values. We expect the two to be roughly tied where the harm is visible from the request alone. That split is the test of the idea.

Core Claim

Context-dependent safety for code agents is best learned as executable guards checked against live program state, not as a natural-language verdict issued before the code runs. The difference should be largest for small students, whose code reasoning is least reliable.

Three lines of evidence motivate this claim. Executable Code Actions Elicit Better LLM Agents introduces CodeAct, where agents act by emitting executable Python. It shows code actions beat JSON or text actions on 12 of 17 models, and it also documents agents assuming what a dataframe holds without inspecting it. Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics finds that code-assisted reasoning is often logically unsound even when the final answers are right. Together these say that any safety judgment resting on the model's mental simulation of its own code is fragile. Is Safety Standard Same for Everyone? introduces U-SAFEBENCH, which tests whether models refuse requests that are harmless in general but hazardous for a specific user profile. On this benchmark, 20 models given the profile average only 14.1% user-specific safety. Two-step text chain-of-thought raises the average from 17.8% to 32.7%, but it cuts helpfulness from 86.7% to 78.5%. Text reasoning therefore helps, but it is blunt.

The guard design also answers a tension in the literature. RedCode runs code agents in live Docker sandboxes and finds they scrutinize risky operations more when those operations appear as code than when they are described in prose. That supports locating safety in code. The unsoundness results cut the other way. Guards reconcile the two findings: the model writes the safety condition, which is short and local, while the interpreter supplies the facts the condition is checked against.

How This Differs From the Closest Work

The nearest systems are VeriGuard and SEVerA.

VeriGuard has a large model write a policy function and formal contracts offline, from the user request and the access policy. A program verifier, Nagini, proves the policy satisfies the contracts, and each action is checked against that verified policy at runtime. SEVerA builds self-evolving agents in Dafny, a programming language with built-in proof checking. Each model call is wrapped in a rejection sampler with verified fallbacks, and SEVerA reports 0% constraint violations at a 1.9–2.5x runtime cost.

Both systems place safety in an external, mostly static artifact built by a large model. Neither studies small students. Neither handles a judgment that depends on a semantic, per-user condition being evaluated over data the agent has just computed. We put the check inside each action, generated by the acting model itself. We enforce only structure: an operation must be covered by a guard. We do not require proof. The whole behavior is distilled into compact models.

A second neighbour is Safety Reasoning with Guidelines (SRG). It distills GPT-4o safety rationales into LLaMA-3.1, cutting out-of-distribution jailbreak success by 46.3% at 8B relative to plain refusal training. Its action space is conversational text only. We reuse its distillation recipe as our main competitor, not as our method.

The distillation machinery comes from Distilling LLM Agent into Small Models with Retrieval and Code Tools. That work trains 0.5B–7B code-tool agents from a 32B teacher. It introduces two techniques we reuse: making the teacher reason before acting, and sampling several actions at test time to discard ones with invalid syntax. It never evaluates safety.

Research Program

We would build a small runtime around a CodeAct-style interpreter. Each task comes with a structured context object describing the user's role, expertise, age band and health conditions, the deployment domain, and the applicable norms. Consequential operations go through a small set of named effect sinks: sending data outward, writing or deleting, executing system commands, and returning advice to the user. An agent passes a sink only by first calling guard(condition, reason). Here condition is a Python expression over the context object and the runtime variables in scope, for example guard(not (ctx.user.diabetic and plan.sugar_g > ctx.limits.sugar), ...). A sink with no guard covering it is blocked. A failed guard returns a structured refusal, or alternatively a safer branch the agent wrote with ordinary if/else.

A short list of universal prohibitions sits beneath every guard and cannot be overridden by any context. Examples are malware families and harms involving child abuse. This follows the point in CASE-Bench that some harms are unacceptable under any framing.

Teacher trajectories come from Qwen2.5-32B. The teacher is prompted with the sink API and asked to reason before acting. Trajectories are kept only if two conditions hold in the sandbox. First, the task outcome must be correct. Second, the safety outcome must match the gold label: blocked when it should be, completed when it should be. Students are Qwen2.5 at 0.5B, 1.5B, 3B and 7B, trained by supervised fine-tuning on these filtered trajectories. An optional second stage applies group-relative reinforcement learning, the GRPO method used for contextual-integrity reasoning in Contextual Integrity in LLMs via Reasoning and Reinforcement Learning. That paper's rewards rely on keyword checks. Our reward is instead read from the sandbox's actual effects. This removes a weakness that paper's design leaves open.

No benchmark tests context-dependent safety where the actions are code, so we would construct one, which we call CtxCode, from existing sources. User profiles and hazards come from U-SAFEBENCH. Safe and unsafe contexts for the same instruction come from CASE-Bench. Risky system operations come from RedCode-Exec, re-framed under privileged and unprivileged roles. Every task is executable against mock APIs: a patient database, a nutrition database, a file system, and an outbound mail endpoint.

Tasks are labelled along one axis that matters most. In the request-visible split, the hazard is apparent from the instruction plus the profile. In the value-dependent split, the hazard appears only in data retrieved or computed during execution. Each split contains matched benign twins, which lets us measure over-refusal. We would also evaluate on EICU-AC and Mind2Web-SC for role-based access control, and on M3ToolEval for plain task utility.

Baselines

The naive reference point is the student distilled exactly as in Distilling LLM Agent into Small Models with Retrieval and Code Tools. It has the same teacher, the same data volume and the same code tools, but no safety supervision and the context object pasted into its prompt. It sets the floor for safety and the ceiling for unconstrained helpfulness. Any method must keep task success within a few points of it on benign tasks and M3ToolEval.

The strongest learned competitor is text safety reasoning distilled SRG-style into the same students. The teacher writes a reflection and a safety verdict about the user and the request before writing unguarded code, and the student learns to do the same. It is the strongest competitor because it is the method that has worked for text, and it uses the same teacher and context. For fairness, it receives the same number of filtered teacher trajectories, the same tuning budget, and the same optional reinforcement-learning stage with the same sandbox reward. The closest reported number for this family is from Safety Reasoning with Guidelines. On LLaMA-3.1-70B, SRG cut PAIR attack success from 94% to 4%, and at 8B it cut average out-of-distribution attack success by 46.3% versus refusal training. On user-specific safety, the best reported text-reasoning result is the U-SAFEBENCH two-step chain-of-thought average of 32.7%, with Claude-3.5-sonnet at 76.7%. No result has been reported for this method family on code actions. Guards beat it if, at equal parameter count, they score higher on the value-dependent split. They must also not lose on the request-visible split or on benign helpfulness, and must use no more generated tokens per task.

The external-classifier baseline runs Llama-Guard-3-8B over every code action, together with the context object, and blocks flagged actions. CASE-Bench reports that this classifier reaches 54.1% accuracy, or 60.1% when context is provided. U-SAFEBENCH reports dedicated safeguard models near 1% on user-specific safety. We run it on the same CtxCode tasks with the same students generating the actions.

As a heavy-machinery reference we include VeriGuard with its large teacher on EICU-AC and Agent Security Bench. VeriGuard reports perfect accuracy on EICU-AC, and on Gemini-2.5-Flash it reports attack success falling from 53.5% to 0% while task success rose from 51.87% to 63.25%. We do not expect a 3B student to beat it on static access control. We report how close the student gets, and whether VeriGuard's offline policies can express the value-dependent tasks at all.

Three ablations isolate the key ingredients. In the first, guards are written but not enforced: sinks without a guard are not blocked. This measures how much depends on the interpreter rule rather than the learned behavior. In the second, the guard is evaluated by the model. The student must state the condition's truth value in text before acting, with no runtime evaluation. This removes exactly the "check live values" property. The third removes the universal-prohibition layer, to quantify how much context flexibility costs on non-negotiable harms.

Why not just…?

Why not just put the user profile in the system prompt? U-SAFEBENCH already measured this across 20 models: 14.1% average safety, and GPT-4o at 24.7%. ConVerse, which benchmarks privacy and security in multi-turn agent-to-agent negotiation, shows frontier agents given explicit privacy instructions still leaking 37–88% of private data under pressure. Small students should do no better. Our naive baseline is this approach, so CtxCode will measure the gap directly rather than assume it.

Why not just distill text safety reasoning, as SRG does, and let the agent write code afterwards? This is the real threat. On the request-visible split, we expect it to tie with guards, because the information needed to judge is already in the prompt. The proposal stands or falls on the value-dependent split, where text reasoning must guess what the code will produce. The second ablation, where the model evaluates its own guard, is the sharpest test. If text reasoning matches guards there, the runtime-evaluation story is wrong. In that case we would report guards as merely a format preference and abandon the claim.

Why not just run a context-aware guard classifier over every action? CASE-Bench gives context-aware Llama-Guard only 60.1% accuracy, even on text. A classifier reading code still sees the code, not the values the code will produce. VeriGuard's authors make the same point: pattern-matching guardrails do not model what code actions do to system state. The classifier baseline on the value-dependent split will show whether this holds for code.

Why not just use sandbox permissions or a static access-control list? For role-only constraints, such as "receptionists cannot read diagnoses," a static rule is simpler and provably correct. We expect it to match or beat learned guards on EICU-AC-style tasks, and we will say so. It cannot express "this meal plan is harmful for this user" or "this retrieved row belongs to a minor" without effectively re-implementing semantic judgment. The value-dependent split exists to separate the two. If most realistic hazards turn out to be expressible as static rules, that result itself narrows the scope of our claim.

Why not just use formal verification like VeriGuard or SEVerA? Formal verification gives guarantees we do not, and we treat VeriGuard as an upper reference. Its specifications are written by large models and need human checking. SEVerA reports a 1.9–2.5x runtime cost and depends on a library of verified fallbacks. Neither has been run on small models. The question here is different: what a 0.5B–7B agent can do on its own, at roughly CodeAct's cost.

Proposed Experiments and Decisive Checks

Our metrics are four. Context-dependent safety rate is the fraction of hazardous tasks correctly blocked or redirected. Benign completion rate is the fraction of matched benign twins completed correctly; it is the over-refusal check. Task success on M3ToolEval is general utility. Generated tokens and wall-clock time per task measure overhead. All are reported per student size and per split.

The first decisive check is a difference of differences. The guard students' advantage over the text-safety students should be clearly larger on the value-dependent split than on the request-visible split, at every student size. We expect the effect to grow as students shrink, because smaller models simulate their own code less reliably. A uniform advantage on both splits would suggest guards help only as extra supervision, not through runtime evaluation. The second check is that guard students avoid the helpfulness loss seen in U-SAFEBENCH, where completion fell from 86.7% to 78.5%. A context condition written as code applies to one sink; it does not tilt the whole response toward refusal. The third check is adversarial. We will rerun the ConVerse-style coercion attacks and the RedCode-Exec natural-language rephrasings against guard students. The question is whether guards resist an "optimize the user goal" pressure that pushes text-reasoning agents to argue themselves past their own safety rules. The last check is the guard-validity rate: how often guards are syntactically broken, vacuous (for example guard(True)), or check the wrong variable. Small students are known to break code syntax. If vacuous guards are common, interpreter enforcement provides false comfort. We would then add a cheap static lint that rejects guards not referencing a context attribute, and report results with and without it.