Iterative Plan Verification for Safe Autonomous Agents

16 concepts11 papers1 proposal

When Large Language Models (LLMs) transition from passive conversationalists to autonomous agents that interact with physical environments, operating systems, and medical workflows, the stakes for safety skyrocket. Traditional alignment techniques like Reinforcement Learning from Human Feedback (RLHF) help ensure "verbal safety," but they often fail when agents are required to execute multi-step tool use or complex planning.

As demonstrated by Yuzhi Tang et al. in LM Agents May Fail to Act on Their Own Risk Knowledge, autonomous agents suffer from a distinct "knowing-doing gap." An agent may possess the abstract knowledge that an action is dangerous, yet still generate a hazardous plan during a single forward pass.

To bridge this gap, a powerful paradigm has emerged: Iterative Plan Verification. Instead of relying on an agent to generate a flawless, safe trajectory zero-shot, these frameworks decouple generation from verification. By embedding the planner inside a closed-loop propose-evaluate-correct cycle, systems can catch blind spots, simulate consequences, and refine actions before they ever touch the real world.


The Propose-Evaluate-Correct Architecture

Why does single-pass planning fail? Generating a plan requires an LLM to simultaneously optimize for task completion, syntax, logic, and safety boundaries. When a single model handles all these dimensions at once, safety constraints are frequently overridden by the high-entropy search for a solution.

Iterative plan verification solves this by introducing a strict separation of concerns. The architecture splits the workflow into distinct functional roles:

  1. The Proposer: A primary agent drafts an initial action or trajectory.
  2. The Critic/Inspector: A secondary high-reasoning model or specialized module evaluates the draft against safety rules, ethical guidelines, or operational constraints.
  3. The Corrector: If a risk is flagged, the feedback is fed back to the proposer to rewrite the plan.
 [ Proposer Agent ] ------( Draft Plan )------> [ Critic / Inspector ]
         ^                                              |
         |                                       (Safety Violation?)
         |                                              |
  (Corrective Feedback)                                 v
         |------------------------------------------- Yes / No -------> [ Execution ]

This multi-stage pipeline is highly effective across diverse domains. In the TrustAgent framework developed by Wenyue Hua et al. (TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution; TrustAgent: Towards Safe and Trustworthy LLM-based Agents), a post-hoc trajectory inspection stage evaluates generated plans against a predefined "Agent Constitution." If the inspector detects a violation, it provides targeted feedback, triggering an iterative revision process that eliminates high-risk errors without sacrificing the agent's helpfulness.

A similar dynamic operates in medical consultation. In Healthcare agent: eliciting the power of large language models for medical consultation, Zhiyao Ren et al. implement a multi-stage generation process that moves from a preliminary draft to a risk-based critique, and finally to a modified output. Evaluating its own preliminary thoughts in a separate step allows the system to identify ethical risks and medical errors far more reliably than single-shot generation.


Grounding Critics via Simulation

While text-based critics can catch logical or constitutional violations, high-stakes physical environments require a deeper level of verification. A driving maneuver or a medical treatment plan might sound perfectly reasonable in natural language but fail catastrophically when subjected to the laws of physics or biology.

To overcome this, researchers are grounding the evaluation phase in predictive simulators that act as safety guardrails.

In autonomous driving, Yihong Guo et al. introduce CorrectionPlanner: Self-Correction Planner with Reinforcement Learning in Autonomous Driving, which utilizes a learned safety critic to evaluate proposed actions. By maintaining a "correction trace" of rejected unsafe actions, the model learns to navigate complex action spaces safely.

To catch long-tailed physical failures like collisions or traffic infractions, frameworks like PlanAgent (Zebin Xing et al. in PlanAgent: A Multi-modal Large Language Agent for Closed-loop Vehicle Motion Planning) and LangDriveCTRL (Francesco Pittaluga et al. in LangDriveCTRL: Natural Language Controllable Driving Scene Editing with Multi-modal Agents) integrate simulation-based reflection modules. These modules physically simulate the proposed vehicle trajectories before execution. If the simulation predicts a collision or a rule violation, it triggers a "rethinking" loop to refine the motion path.

[ Generative Planner ] ---> (Trajectory) ---> [ Forward Simulator ]
         ^                                           |
         |                                   (Collision Detected?)
         |                                           |
  (Refinement Prompt)                                v
         |---------------------------------------- Yes / No -------> [ Real-World Drive ]

This closed-loop simulation strategy is equally vital in clinical settings. Ayan Banerjee et al. demonstrate in Framework for Developing and Evaluating Ethical Collaboration Between Expert and Machine that passing generative treatment plans through high-fidelity physiological simulators allows the system to test novel strategies safely. If the simulator predicts an adverse patient outcome, the feedback is used to iteratively refine the proposal before any human implementation occurs.


Granularity of the Safety Loop

Iterative plan verification can be integrated at different temporal granularities depending on the latency requirements and risk profile of the application.

Post-Hoc Trajectory Verification

The entire plan or sequence of actions is generated and reviewed as a complete package. For example, the Windows OS navigation agent UFO, developed by Chaoyun Zhang et al. in UFO: A UI-Focused Agent for Windows OS Interaction, evaluates the outcomes of executed UI actions against expected results. If an error or high-risk action is detected, it dynamically revises its strategy or pauses for human confirmation.

Step-Wise Monitoring

The critic evaluates the agent's internal reasoning and planned actions at every discrete step of execution. In Athena: Safe Autonomous Agents with Verbal Contrastive Learning, Tanmana Sadhu et al. deploy a real-time secondary model to intercept and correct hazardous decisions at each step, ensuring that the agent never takes an unverified action in the environment.

Decoder-Level Intervention

The most granular approach intervenes during the token generation process itself. Xinyi Zeng et al. propose this in ROOT DEFENCE STRATEGIES: ENSURING SAFETY OF LLM AT THE DECODER LEVEL. By performing step-by-step assessments and corrections at the decoding level, their framework prevents the model from completing an unsafe thought. The authors demonstrate that this micro-level iteration is less prone to over-correction than evaluating a finished macro-output.


Proactive Defense and Lesson Learning

A common failure mode of iterative refinement is that agents can fall into self-reinforcing error cycles, where they repeatedly propose variations of the same flawed plan. To break these loops, modern verification frameworks incorporate memory-based proactive defenses.

In A-MemGuard: A Proactive Defense Framework For LLM-Based Agent Memory, Qianshan Wei et al. introduce a "lesson memory" repository containing known failure modes and security risks (such as memory injection attacks). Before an action is finalized, the system cross-references the proposed trajectory with these past lessons. If the action aligns with a known risk, the agent enters a deliberation phase to proactively rewrite its plan, ensuring that the iterative loop is guided by historical failures.

By separating generation from evaluation, grounding critics in simulation, and varying the granularity of intervention, iterative plan verification transforms LLMs from fragile, zero-shot planners into resilient, self-correcting autonomous agents.

Key Foundational Work

TrustAgent: Towards Safe and Trustworthy LLM-based Agents through Agent Constitution (2024) — contributed 2 concept elements to this direction.
TrustAgent: Towards Safe and Trustworthy LLM-based Agents (2024) — contributed 2 concept elements to this direction.
Healthcare agent: eliciting the power of large language models for medical consultation (2025) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_plan_refinement_0709d293,
  title = {iterative plan refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-plan-refinement/rdr_0709d293ee8f43af90ae4ac761731e74},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_iterative_plan_refinement_0709d293_v1,
  title = {iterative plan refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-plan-refinement/rdr_0709d293ee8f43af90ae4ac761731e74/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake