Safeguarding Agentic AI in Radiology Workflows: A Multimodel Evaluation of Rule-based and Large Language Model-based Verification

Semil Eminovic, Robin Schmidt, Bogdan Levita, Maximilian Lindholz, Tobias Penzkofer, Jawed NawabiarXiv 2026 · arXiv

TL;DR

This paper evaluates how "agentic" AI—LLMs that can use software tools to act on medical records—performs in radiology workflows. The researchers found that while layered safeguards (deterministic rules and an LLM-based checker) can reduce high-severity clinical errors by about one-third, they cannot yet prevent them entirely, especially when the AI is faced with plausible but misleading information.

What Problem Does This Paper Address?

We are moving from LLMs that simply answer questions to "agentic" AI that can plan and execute actions, such as writing summaries or updating follow-up recommendations in a clinical information system. In medicine, the stakes for this shift are massive: a wrong answer can be ignored, but a wrong action writes false information into a patient's permanent record.

The authors identify a specific threat they call "ordinary integrity problems." Unlike malicious prompt injections (where an attacker tries to hijack the AI), these are everyday clinical situations where information is plausible but misleading—such as an outdated report or a note from a referrer that incorrectly identifies which side of the body a finding is on. The paper investigates whether automated "guardrails" can catch these errors before they are committed to the record.

Methods

The researchers built a simulated radiology environment and tested six different LLM agents (including Claude 3.5 Sonnet, GPT-4-class models, and open-weight models like Gemma and Qwen).

The core of the study is the Safeguard Gate, a multi-layered verification system placed between the AI's proposed action and its execution:

  1. Permissions Only: The baseline, checking only if the agent has the right to perform the action.
  2. Deterministic Rules: 11 hard-coded checks (e.g., "Does the cited source actually belong to this patient?" or "Is this the most current version of the report?").
  3. LLM Checker: A separate, locally-run LLM (Gemma-4-31B) that reviews the proposed action against all available sources to check for contradictions, omissions, or unsupported claims.
The study design and agent workflow, showing the "Safeguard Gate" intervention.

Proposed actions that failed these checks were sent back to the agent for up to two revisions.

Datasets

The study used 48 synthetic, radiologist-reviewed base scenarios covering three modules:

  • M1: Finding transfer and communication.
  • M2: Tumor board preparation.
  • M3: Follow-up of incidental findings.

These were expanded into 91 variants, including 43 "misleading-information" variants where a specific piece of false but realistic data was inserted into an external document. An independent, deterministic evaluator (not an LLM) scored the final outputs against a "gold standard" reference.

Key Insights and Ideas

  • "Misjudged, not missed": In 83% of episodes where the AI made a high-severity error, it had actually opened and read the misleading document. The error wasn't a failure of retrieval; it was a failure of reasoning—the AI "believed" the misleading information over the correct data.
  • Layered Defense Works: Deterministic rules and LLM checkers are complementary. Rules are great at catching formal errors (wrong patient ID), while the LLM checker is better at catching semantic errors (contradicting clinical facts).
  • The Cost of Safety: Most blocked actions (64%–68%) were "unnecessary blocks," meaning the action wouldn't have actually caused a high-severity violation. While this didn't hurt task success in the simulation (because the AI just revised its work), in a real-world setting, this would create "revision fatigue" for human supervisors.
  • Weak-to-Strong Supervision: Interestingly, using a smaller, local model as a "checker" for larger, more powerful "agent" models was effective, suggesting that verification might be a "simpler" task than agentic planning.

Key Results

The study analyzed 1,638 total "episodes." * Baseline Errors: With only basic permissions, agents committed at least one high-severity violation in 19.4% of cases. * Impact of Safeguards: Adding deterministic rules dropped the error rate to 16.7%. Adding the LLM checker on top of that brought it down to 13.2%. * Misleading Info: Errors were significantly higher in misleading variants (25.0%) compared to clean ones (14.9%). * Task Success: Crucially, adding these safeguards did not decrease the agents' ability to successfully complete their tasks.

Risk differences showing how rules and LLM checkers reduced high-severity errors across different models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Synthetic & Text-Only: The study used synthetic data and did not include actual medical images, which are central to radiology.
  • Language: The study was conducted entirely in German.
  • Human-in-the-loop: The simulation didn't account for how a human radiologist might interact with (or be biased by) these agentic systems.

Future work should focus on "content-aware" safeguards that can judge if a plausible statement is clinically correct, rather than just formally consistent. The authors conclude that while safeguards are a necessary building block for autonomous agents, they are not yet a replacement for human oversight.

Summary updated

Abstract

Purpose: To estimate how often tool-using large language model (LLM) agents record clinically relevant false or unsupported information in simulated radiology documentation and how much safeguards applied before each action reduce these errors. Materials and Methods: In this simulation study (October 2026), 48 synthetic, radiologist-reviewed base scenarios covered finding communication, tumor board preparation and follow-up management. Forty-three also had a misleading-information variant (91 variants). Six LLM agents (Claude Sonnet 5.5, GPT-6.1-Sol, GLM-5.3, MiniMax M3, Gemma-4-31B, Qwen3.8-27B) completed each variant with permissions only, deterministic rules, and rules plus a locally run LLM checker (1638 episodes). The primary outcome was at least one executed high-severity violation per episode. Risk differences were estimated per agent at base-scenario level and, exploratorily, pooled across agents, with analyses of document access and blocks. Results: With permissions only, 7.3%-27.1% of episodes had a high-severity violation (pooled across agents, 19.4%). Pooled, it was 16.7% with rules (-2.8 percentage points; 95% CI: -6.1, 0.5) and 13.2% with rules plus the LLM checker (-6.2; 95% CI: -9.5, -3.0). Adding the LLM checker to rules changed it by -3.5 (95% CI: -6.3, -0.7). No difference between agents was detected (P = .43), and task success did not decrease detectably. In misleading variants, agents opened the altered document in 83% of episodes regardless of error, and 58% of erroneous statements cited it. Most blocked actions (64%-68%) would not have caused a violation and were returned for revision. Conclusion: Layered safeguards reduced but did not prevent high-severity errors. Errors after misleading information reflected misjudged rather than missed information.
BibTeX
@misc{eminovic2026safeguarding,
  author = {Semil Eminovic and Robin Schmidt and Bogdan Levita and Maximilian Lindholz and Tobias Penzkofer and Jawed Nawabi},
  title = {{Safeguarding Agentic AI in Radiology Workflows: A Multimodel Evaluation of Rule-based and Large Language Model-based Verification}},
  year = {2026},
  eprint = {2611.11541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11541},
}
Figure