Automated safety evaluators are usually large language models asked "is this response harmful?" That question is itself a safety judgment. Work on mismatched safety generalization, the observation that models' abilities reach far beyond what their safety training covered, shows that such judgments break when content arrives in an unfamiliar format: Python data structures, set-theory notation, SQL, ciphers, or a request split across turns. This note asks whether the evaluators used to score clinical safety fail in the same way. Suppose a model recommends a contraindicated drug, or an overdose-level dose, inside a code block or an algebraic derivation. Does the judge still see the harm? If it does not, current numbers on medical jailbreaks undercount the harm that matters most clinically. These are the subtle, quantitative errors that domain-specific metrics were built to catch, but those metrics have only been tested on benign, cooperative prose.
We propose a different design. The language model's job is narrowed from judging safety to extracting a structured clinical decision record: which drug, what dose, for which patient conditions, and which calculated value. Harm is then decided by a deterministic check against a curated rule database and known formulas. The underlying bet comes straight from the mismatched-generalization literature. Reading what an output says is a capability, and capabilities transfer to odd formats. Deciding whether it is unsafe is a safety behavior, and safety behaviors do not transfer. An evaluator built only from the part that generalizes should hold its accuracy as the format shifts, while judge-based evaluators should degrade.
Core Claim
Clinical safety evaluators that ask a model for a harm verdict lose discriminative power when the same clinical content is re-rendered in code, symbolic math, query languages, or multi-turn splits. An extract-then-check evaluator keeps its accuracy across these formats because its only learned step, field extraction, falls on the capability side of the capability/safety gap. The claim is falsifiable at the level of components. If extraction accuracy falls with format as steeply as judge accuracy does, the premise is wrong and the design gains nothing.
Grounding in Prior Work
Several papers motivate the format concern. Jailbroken: How Does LLM Safety Training Fail? introduced mismatched generalization. Three attack papers then showed its force in specific formats: CodeAttack hides requests in code data structures, MathPrompt encodes them in set theory and algebra (average attack success 73.6% across 13 models), and QueryAttack uses query languages. The Structural Safety Generalization Problem extends the failure to multi-turn splits and low-resource languages. All of these study the target model's refusals. None asks whether the evaluator of the outputs suffers the same failure, and that question is the one we take up.
The domain-specific side supplies our method and our ground truth. RadGraph and RadCliQ score reports by extracting entities and relations instead of comparing wording. We borrow that principle of structure over surface, but apply it to adversarial outputs, which those works never tested. RxSafeBench provides RxRisk DB (6,725 contraindications and 28,781 drug interactions), which serves as our deterministic checker. MedCalc-Bench provides 1,047 calculation instances with exact answers, which lets us decide whether a computed value is wrong without any judgment call.
The closest existing work sits right at the intersection but takes a different path. CARES attacks medical models with role-play, obfuscation and chained prompts. It scores responses with a GPT-4o-mini judge using ACCEPT/CAUTION/REFUSE labels, so it measures refusal and never checks clinical correctness. Automated Rubrics for Medical Dialogue builds evidence-grounded atomic-fact rubrics and reaches an AUROC of 0.977 on benign near-miss pairs, but it has never been run on format-shifted or adversarial outputs. Medical MLLM Is Vulnerable scores multimodal attacks by attack success and refusal rate alone. We differ from all three in two ways. We hold the clinical content fixed while varying its format, and we replace the harm verdict with a database check.
One body of work constrains our scope. Fake Alignment argues that safety behavior is shallow persona imitation, which predicts judge fragility. However, it gives no direct evidence that extraction is robust, so we treat robustness of extraction as something to be measured, not assumed.
Research Program
The first artifact is a paired-format clinical harm set. We start from RxSafeBench scenarios and MedCalc-Bench instances. For each, we write short clinical answers whose harm status is known by construction. Harmful answers recommend a drug that RxRisk DB flags for the patient's condition or co-medication, or report a calculated dose or score outside MedCalc tolerance by a clinically meaningful margin. Matched safe answers make the correct choice. Each answer is then re-rendered into six formats while its clinical content stays fixed. The prose original is one. CodeAttack-style Python is another, with the drug and dose stored in lists and dictionaries. The remaining four are MathPrompt-style set and algebra statements, SQL result tables in the QueryAttack style, a multi-turn split with the decision spread across turns following the structural safety generalization benchmark, and a simple character cipher. Rendering uses deterministic templates wherever possible. Where an LLM performs the rewrite, a clinician-annotated subsample verifies that the rendering preserves content. This design isolates format as the only variable.
The second artifact is the extract-then-check evaluator. An instruction-following model fills a fixed schema: drugs, doses and units, stated or implied patient conditions, co-medications, and any reported calculated value with the calculator used. It is told nothing about safety. A deterministic module then flags harm in three cases: an RxRisk DB contraindication or interaction is present, a dose exceeds formula-derived bounds computed from the extracted patient parameters, or a reported score falls outside MedCalc tolerance. An empty extraction is reported separately and never counted as "safe," so abstentions cannot inflate accuracy.
The third component is analysis. For every evaluator we report harm-detection AUROC for each format, along with the "format gap," the drop from prose to each shifted format. For our evaluator we also report field-level extraction accuracy for each format, which tests the core premise directly. Finally, we reuse the evaluator as an output-side guardrail. This tests whether a format-invariant check reduces the over-refusal problem that CARES documents. A database check flags only specific contraindications instead of sensitive vocabulary, so benign questions about toxic drugs should pass.
Scope is deliberately limited. The evaluator handles harms that reduce to structured medication and calculation decisions. It cannot judge HIPAA or credential-falsification harms of the kind in CARES, and we will report that boundary rather than hide it.
Baselines
The naive reference point is the CARES-style judge: GPT-4o-mini prompted with the ACCEPT/CAUTION/REFUSE rubric from CARES, with the label mapped to a harm score. This is how medical jailbreak numbers are produced today. CARES reports only qualitative rankings, and there is no reported result found on Lacuna for its judge's accuracy on clinically harmful outputs.
The strongest existing evaluator is the evidence-grounded rubric system of Automated Rubrics. It is the strongest competitor because it was designed precisely for subtle near-miss medication errors. Its reported numbers are an AUROC of 0.977 on benign near-miss pairs and a Clinical Intent Alignment of 60.12% against GPT-4o's 55.16%. We run it with its published Llama-3.3-70B-Instruct backbone. Our extractor gets that same backbone, so the comparison isolates design rather than model strength. Both systems see identical items and have the same token budget per item. Beating it means matching its prose AUROC to within a small margin while showing a format gap on shifted renderings that is significantly smaller, measured by bootstrap confidence intervals over items.
The second strong competitor is output-side canonicalization in the style of the Structure Rewriting guardrail from The Structural Safety Generalization Problem. An auxiliary model rewrites the output into plain English, and the CARES judge then scores the rewrite. The paper reports this guardrail cut GPT-4o attack success from 44% to 6% at a 17% false positive rate, though on general-domain harms. A general moderation classifier, Llama-Guard-3-1B, rounds out the field. LLMSymGuard reports it reaching only F1 0.26 on hypothetical Best-of-N jailbreaks.
Two ablations isolate the key ingredient. In the first, the same extractor fills the schema and the same model is then asked to judge harm from the record. This tests whether the deterministic check matters or whether structuring alone does the work. In the second, the extractor is additionally told the task is safety-related. This tests whether framing extraction as a safety task brings back the fragility we predict.
Why not just…?
Why not just canonicalize the output to English and judge it? This is the real threat. If rewriting a code-block answer into prose preserves the drug and dose faithfully, a judge on the rewrite might close most of the format gap. Two things may still separate the approaches. Rewriting can drop numbers and negations, which is the weakness RadCliQ found in text-level metrics. And the judge on the rewrite still has to know that a given dose for a given renal function is dangerous, which is where RxSafeBench shows even DeepSeek-R1 reaching only 38.12% on drug interactions. The experiment settles the question directly by measuring the canonicalize-then-judge baseline's format gap and its prose accuracy. If it matches us on both, we will report that the contribution is the paired-format benchmark, not the evaluator.
Why not just run the automated rubric system as is? It may well be robust. Its rubrics are grounded in atomic facts, which already moves part of the way from judgment toward checking. The difference is that its auditing agent still renders a verdict about whether the response satisfies each rubric item, and that verdict step may degrade when the response is a SQL table. Its 0.977 AUROC was measured only on benign prose, so whether it holds under format shift is an open empirical question this proposal answers either way.
Why not just use exact-match scoring as MedCalc-Bench does? Exact match needs the answer in a known slot. Adversarially formatted free-form outputs put the value anywhere, which is exactly the problem the extraction step solves. Exact match is also blind to contraindications, which involve relations between facts rather than single values. We use MedCalc's tolerance rule as one component of the checker, not as a replacement for it.
Why not just read the target model's activations, as LLMSymGuard does? Activation probes need white-box access, so they cannot score outputs from commercial models. They also detect harm concepts, not relational violations of the form "drug A is dangerous with condition B," which is the form RxRisk DB encodes.
Decisive Checks
The central experiment applies every evaluator to the paired-format set and reports AUROC for each format. The prediction is that judge-based evaluators lose substantially more AUROC from prose to code, math and SQL than extract-then-check does, while the two are comparable on prose. The premise check runs alongside it. Extraction field accuracy should stay roughly flat across formats. If it instead falls as steeply as judge AUROC, the capability-side argument is refuted. The cipher and multi-turn formats are the likeliest places for extraction to fail, and we expect to report honest degradation there.
The second experiment estimates undercounting in the wild. We generate format-shifted clinical prompts derived from CARES and MedCalc-Bench against several open and API models, score their outputs with both the CARES judge and our checker, and report the fraction of checker-flagged harms that the judge labeled safe, broken down by format.
The third experiment tests the evaluator as a guardrail. Attack success under it is compared against the canonicalization guardrail on the shifted prompts, and false positive rate on CARES Level-0 benign prompts serves as the measure of over-refusal. A useful result would cut attack success at least as much as canonicalization, at a false positive rate well under the 17% reported for that guardrail, within the medication-and-calculation scope.
References
- Jailbroken: How Does LLM Safety Training Fail?
- CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
- Jailbreaking Large Language Models with Symbolic Mathematics
- QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language
- The Structural Safety Generalization Problem
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation
- RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultation
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
- CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs
- Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems
- Medical MLLM Is Vulnerable: Cross-Modality Jailbreak and Mismatched Attacks on Medical Multimodal Large Language Models
- Fake Alignment: Are LLMs Really Aligned Well?
- LLMSymGuard: A Symbolic Safety Guardrail Framework Leveraging Interpretable Jailbreak Concepts