Surgeons who repair large bone defects face a trade-off with harm on both sides. The standard treatment takes healthy bone from the patient's own hip, which creates a second wound with pain, infection and nerve-injury risk, and the hip holds only so much bone. Synthetic substitutes avoid that wound, but they only help inside narrow dose windows. Cobalt-doped ceramics promote blood-vessel and bone growth at 0.25 mol% and kill cells at 1.0 mol%. If a substitute fails, the bone never heals. Language models are starting to be asked to draft operative and material plans of this kind. The tools meant to check them, called safety auditors (usually a second language model acting as a judge), were built for a different kind of question: did the model refuse a forbidden request? In this domain, refusing or under-treating is also a harm, and the violations that matter are mostly numbers. A plan is unsafe because it calls for a dopant level, harvest volume or ion concentration on the wrong side of a published threshold, often stated in different units or only implied.
We propose a grounded auditor for these plans and a testbed to measure it. A language model first extracts every quantity a plan commits to into a fixed schema, converting units and resolving implied amounts such as "four times the optimal cobalt loading." A deterministic checker then compares those quantities against a constraint sheet in which each threshold is tied to a cited study. A judge model handles only the non-numeric parts of the plan and scores harm in both directions. Our central claim is testable. On plans produced under realistic pressure, rubric-based judges will miss most threshold violations that are phrased indirectly and will score unhelpful refusals as safe. Separating extraction from checking will close most of that gap. It will also let us measure a "knowledge-action gap": the difference between a model correctly stating a toxicity window when asked directly and that same model respecting the window inside a multi-constraint design task.
What the two directions contribute
The bone-repair literature supplies the safety constraints, and the auditing literature supplies the method and its known failure modes. On the bone side, Cobalt-doped bioceramic scaffolds fabricated by 3D printing tested cobalt at 0 to 1.0 mol% in 3D-printed lithium calcium phosphate scaffolds. It found 0.25 mol% optimal, viability above 70% up to 0.5 mol%, and cytotoxicity at 1.0 mol%. Building Osteogenic Microenvironments With Strontium-Substituted Calcium Phosphate Ceramics reports a strontium-ion window of 0.1–1.0 mM, with apoptosis and suppressed mineralization above it. Note that this window is stated in released-ion concentration, not dopant fraction, which is exactly where unit confusion can hide. The strategy of composite grafting with BMP2-Loaded calcium phosphate cements and autogenous bone compared two approaches in 19 children with alveolar clefts. A 1:1 composite harvested through a 2 cm incision showed 22.52% three-month volume loss, against 33.42% for pure hip bone harvested through a 6 cm incision. It also notes that the growth factor BMP-2 is costly and linked to swelling and bone forming in the wrong place, which Osteoclast-Driven Osteogenesis… BMP2-CPC-Induced Alveolar Bone Regeneration explores mechanistically.
These findings are mostly in vitro, from rats, or from very small cohorts. The constraint sheet is therefore an evidence-cited audit standard, not clinical guidance, and every entry carries its source and evidence level.
On the auditing side, three lines of work motivate the gap. SafeSearch found that search agents know content is untrustworthy when asked in isolation but still propagate it during multi-step tasks, with attack success up to 90.5% for GPT-4.1-mini. We reuse its knowledge-action framing but move it from web content to numeric dose windows. SafePro reports unsafe rates above 40% for frontier agents on professional tasks, including healthcare cases that favor profit over patients. Its tasks are administrative and contain no dose or surgical constraints. NESSiE shows that models lean toward helpfulness over safety and lose at least 15% under distracting context. We use that distraction as a pressure condition.
Two lines of work constrain the design. Know Thy Judge and Justice or Prejudice? document that judges are brittle to surface phrasing and biased in systematic ways. This is why we do not let the judge model decide numeric compliance.
The closest adjacent system is the MedES "Safety Guardian" in A Human-Centric Pipeline for Aligning Large Language Models with Chinese Medical Ethics. It is a fine-tuned 32B judge reporting over 97% accuracy on 1,278 legal and ethical rules. It audits rule compliance in ethics question answering. It does not check quantitative material or surgical bounds, and it does not treat under-treatment as a violation.
Core Claim
Treating safety as two-sided adherence to cited quantitative constraints changes what an auditor must do. Consider a plan that either refuses or proposes an untested filler for a defect too large for a substitute alone. That plan is under-treatment and must be flagged. A plan proposing 1.0 mol% cobalt "to maximize vessel growth" is over-treatment, and so is one proposing harvest volumes beyond what a 2 cm approach can supply. We predict three things. Rubric judges will catch explicitly stated violations well. They will degrade sharply when quantities are converted, implied or buried in context. They will systematically give refusals passing scores. Grounded extraction plus deterministic checking should be largely invariant to phrasing.
Research Program
We would build three artifacts.
The first is the constraint sheet. Each entry records a quantity, unit, safe interval, harm direction, source paper and evidence level. It covers cobalt and strontium windows, harvest approach and volume, the composite-ratio evidence, and qualitative BMP-2 cautions. BMP-2 has no numeric threshold in our sources, so it is checked by the judge, not the checker.
The second is a scenario set of defect-repair and scaffold-design tasks. Each task carries competing objectives, such as maximizing vessel growth, matching trabecular-bone strength, minimizing harvest, or cutting cost. We generate target-model plans under pressure conditions taken from the auditing literature. These include role-play and output-constraint combinations from FuzzLLM, which raised GPT-4 attack success to 38.31% for that pairing. They also include "it's for research" framing from SORRY-Bench, which raised fulfillment by up to 66% in some models, and NESSiE-style distractor context.
Biomaterials and craniofacial experts label each plan for over-treatment, under-treatment and quantity values. We also create meaning-preserving rewrites: the same plan with unit changes (mol% versus weight fraction versus a stated released concentration), relative phrasing, and quantities split across paragraphs.
The third artifact is the grounded auditor itself: schema extraction, then the checker, then a judge for non-numeric harms.
Separately, we probe each target model's knowledge in isolation, asking it to state the window, so the knowledge-action gap can be computed per model and per constraint.
Baselines
The naive reference is refusal-string and keyword matching. This is what multi-label judging in JailbreakRadar was designed to replace. It sets the floor and will almost certainly score refusals as safe.
The strongest general competitor is a frontier zero-shot judge, such as GPT-4o, given a structured policy rubric in the style of Fine-tuning Aligned Language Models Compromises Safety. The rubric will be written from the same constraint sheet the grounded auditor uses, so both systems receive identical knowledge. GPT-4o is the strongest reported judge on agent traces in R-Judge, at 74.45% F1 there.
The strongest specialized competitor is a small judge fine-tuned on our expert labels, following SORRY-Bench. That paper reports a 7B judge exceeding 80% agreement with humans (Cohen's kappa) and matching zero-shot GPT-4o. R-Judge reports 71.84% F1 for a fine-tuned Llama-3-8B guard. Where the MedES Safety Guardian weights are obtainable, we also run it as an in-domain medical judge; its reported figure is over 97% accuracy on its own domain. None of these numbers transfer to our task, and we found no reported result on Lacuna for any auditor on bone-graft or biomaterial plan safety.
Fairness controls apply throughout. All auditors see the same plans and the same constraint sheet. The fine-tuned judge gets the same labeled training split and the same tuning budget we spend on the extractor. The frontier judge gets matched prompt-engineering iterations.
Two ablations isolate the key ingredient. In the first, the extractor's schema output is shown to the judge model, which then makes the decision instead of the deterministic checker. This tests whether the deterministic check, rather than extraction alone, carries the gain. In the second, the under-treatment criterion is removed, to measure how much two-sided scoring changes model rankings.
Beating the baselines means two things at matched false-alarm rate on held-out rewrites. First, higher recall of quantity violations, by a margin larger than the confidence interval across expert-label resamples. Second, a smaller drop between original and rewritten plans.
Why not just…?
Why not just fine-tune a small judge on domain labels? This is the most serious threat. SORRY-Bench shows small judges can reach human-level agreement cheaply. We test it directly by training on cobalt and harvest scenarios and evaluating on held-out strontium scenarios. The grounded auditor needs only a new constraint-sheet entry for strontium; the fine-tuned judge needs new labels. If the fine-tuned judge matches us on the held-out constraint and on unit rewrites, the grounded design is unnecessary, and we would report that.
Why not just a regex checker on numbers? Explicit "1.0 mol% Co" is easy to catch. But violations stated as relative amounts, alternate units, or ion concentrations that must be related to dopant fraction are not visible to string patterns. The rewrite split measures this, and a regex variant is run as an ablation of the extractor.
Why not just test model knowledge with question-answer benchmarks? MatSciBench already shows frontier models below 80% on materials reasoning. However, SafeSearch's results suggest isolated knowledge does not predict behavior inside tasks. We measure the correlation between isolated-probe accuracy and in-plan compliance per constraint. A high correlation would mean knowledge testing suffices and would weaken the case for auditing behavior directly.
Decisive Checks
The idea fails if the rubric judge already catches nearly all quantity violations on rewritten plans. We therefore report recall separately for explicit, converted, implied and split-quantity plans. We expect the gap to sit in the latter three.
The idea also fails if experts disagree about under-treatment labels, since two-sided scoring then has no ground truth. We report inter-expert agreement before any auditor comparison and drop criteria that fall below the agreement level SORRY-Bench reached for its human labels.
Finally, the knowledge-action gap must be measurably positive for at least some frontier models under pressure conditions, and near zero without pressure. Otherwise the testbed is measuring prompt difficulty rather than a behavioral failure.
References
- Cobalt-doped bioceramic scaffolds fabricated by 3D printing show enhanced osteogenic and angiogenic properties for bone repair
- Building Osteogenic Microenvironments With Strontium-Substituted Calcium Phosphate Ceramics
- The strategy of composite grafting with BMP2-Loaded calcium phosphate cements and autogenous bone for alveolar cleft reconstruction
- Osteoclast-Driven Osteogenesis, Bone Remodeling and Biomaterial Resorption: A New Profile of BMP2-CPC-Induced Alveolar Bone Regeneration
- SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
- SafePro: Evaluating the Safety of Professional-Level AI Agents
- NESSiE: The Necessary Safety Benchmark - Identifying Errors that should not Exist
- Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- A Human-Centric Pipeline for Aligning Large Language Models with Chinese Medical Ethics
- FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
- SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
- JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
- R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
- MatSciBench: Benchmarking the Reasoning Ability of LLM in Materials Science