Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking

Haoyu Wang, Christopher M. Poskitt, Jun Sun, Jiali Wei

Abstract

Large Language Model (LLM) agents are increasingly deployed in autonomous roles, yet their stochastic nature poses significant safety risks. Traditional reactive enforcement systems often intervene too late, responding only when a violation is imminent. We present Pro2Guard, a framework that leverages probabilistic model checking to anticipate and prevent safety violations. By modeling agent behavior as a Discrete-Time Markov Chain (DTMC) and applying Probably Approximately Correct (PAC) bounds, Pro2Guard provides a statistically sound method for proactive intervention. Our evaluations in autonomous driving and embodied AI demonstrate that Pro2Guard can predict violations up to 38 seconds in advance and maintain high task success rates through intelligent self-reflection.

Introduction

The transition of LLM agents from sandboxed environments to real-world applications—such as autonomous vehicles and robotic assistants—necessitates rigorous safety guarantees. While existing frameworks like AgentSpec provide rule-based enforcement, they are primarily reactive, triggering only when a violation is imminent or has already occurred. This "kill-switch" approach is often insufficient for complex, long-horizon tasks where a sequence of seemingly benign actions can lead to an inevitable disaster.

Pro2Guard addresses this by shifting the paradigm from reactive to proactive enforcement. By calculating the probability of future safety violations based on current trajectories, the system can intervene early, allowing agents to course-correct before reaching a point of no return.

Methodology

Pro2Guard operates through a structured four-stage pipeline designed to bridge the gap between neural stochasticity and formal verification.

1. Trace Collection and State Abstraction

The process begins by gathering execution traces from the agent within its target environment. To manage the high-dimensional and often unstructured output of LLMs, Pro2Guard abstracts concrete states into symbolic Boolean predicates. For instance, raw sensor data in a driving scenario is mapped to predicates like is_near_pedestrian or is_speeding.

2. Learning the Discrete-Time Markov Chain (DTMC)

These abstracted traces are used to construct a DTMC, representing the agent's behavioral patterns. To ensure the model remains robust against rare events and distribution shifts, the framework employs valid-transition-aware Laplace smoothing. This statistical grounding allows Pro2Guard to offer PAC guarantees, ensuring that the estimated risk is a reliable approximation of the ground-truth behavior.

3. Probabilistic Model Checking

At runtime, Pro2Guard monitors the agent's state transitions. It utilizes Computation Tree Logic (CTL) to perform reachability analysis, calculating the probability that the agent will satisfy safety requirements in the future. If the predicted risk exceeds a user-defined threshold, the system triggers an intervention.

4. Proactive Enforcement Modes

Pro2Guard offers two primary intervention strategies to balance safety with task utility:

  • Self-Reflection: The system provides the LLM with a natural language "risk alert," explaining the potential danger. This prompts the agent to re-evaluate its plan and course-correct.
  • Hard Stop: For critical risks where reflection is insufficient, the system executes an emergency stop or pauses the agent to prevent immediate harm.

Experimental Evaluation

The framework was tested across two high-stakes domains:

  • Autonomous Driving: Utilizing the Apollo platform and μDrive, the system was tasked with identifying traffic law violations and potential collisions using safety rules derived from the LawBreaker specification.
  • Embodied Agents: Using SafeAgentBench, agents performed household tasks (e.g., using appliances) where safety rules (e.g., "do not put metal in a powered-on microwave") were strictly enforced within the ReAct framework.

Results and Discussion

Pro2Guard demonstrated significant improvements over reactive baselines in both safety and utility.

Early Warning and Accuracy

In autonomous driving scenarios, Pro2Guard achieved 100% prediction accuracy for violations. Most notably, it provided warnings between 0.77 and 38.66 seconds before a violation occurred. This lead time is critical for real-world safety systems where mechanical latency (e.g., braking distance) must be considered. It significantly outperformed baselines like REDriver, which struggle with the inconsistent scales of quantitative semantics.

Balancing Safety and Utility

A common drawback of safety enforcement is "safety over-correction," where an agent becomes too timid to complete its task. Pro2Guard’s self-reflection mode mitigated this, enforcing safety on 65.37% of otherwise unsafe trajectories while maintaining an 80.4% task completion rate. In embodied agent tasks, low-threshold settings allowed the system to catch up to 93.6% of unsafe tasks.

Computational and Token Efficiency

The runtime overhead for Pro2Guard is minimal, with model checking taking as little as 5ms, making it viable for real-time robotics. Furthermore, by proactively rejecting trajectories destined for failure, the system reduced LLM token consumption by an average of 12.05%, providing a secondary benefit of cost and latency reduction.

Conclusion

Pro2Guard represents a significant step toward verifiable safety for LLM agents. By combining the flexibility of LLMs with the rigor of probabilistic model checking, it enables a proactive approach to risk management. While the system currently relies on domain experts for predicate definition, future work will focus on automating the discovery of symbolic predicates and extending the framework to multi-agent environments where safety depends on complex interactions between multiple autonomous entities.

Summary updated

Abstract

Large Language Model (LLM) agents exhibit powerful autonomous capabilities across domains such as robotics, virtual assistants, and web automation. However, their stochastic behavior introduces significant safety risks that are difficult to anticipate. Existing rule-based enforcement systems, such as AgentSpec, focus on developing reactive safety rules, which typically respond only when unsafe behavior is imminent or has already occurred. These systems lack foresight and struggle with long-horizon dependencies and distribution shifts. To address these limitations, we propose Pro2Guard, a proactive runtime enforcement framework grounded in probabilistic reachability analysis. Pro2Guard abstracts agent behaviors into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces. At runtime, it anticipates future risks by estimating the probability of reaching unsafe states, triggering interventions before violations occur when the predicted risk exceeds a user-defined threshold. By incorporating semantic validity checks and leveraging PAC bounds, Pro2Guard ensures statistical reliability while approximating the underlying ground-truth model. We evaluate Pro2Guard extensively across two safety-critical domains: embodied household agents and autonomous vehicles. In embodied agent tasks, Pro2Guard enforces safety early on up to 93.6% of unsafe tasks using low thresholds, while configurable modes (e.g., reflect) allow balancing safety with task success, maintaining up to 80.4% task completion. In autonomous driving scenarios, Pro2Guard achieves 100% prediction of traffic law violations and collisions, anticipating risks up to 38.66 seconds ahead.
BibTeX
@misc{wang2025pro,
  author = {Haoyu Wang and Christopher M. Poskitt and Jun Sun and Jiali Wei},
  title = {{Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking}},
  year = {2025},
  eprint = {2508.00500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.00500},
}
Figure