Dependence-Aware Risk Assessment and Optimal Design of AI-Agent Safeguard Systems

Yijia Li, Maochao Xu, Peng ZhaoarXiv 2026 · arXiv

TL;DR

Current AI safeguard systems often fail because they assume different "guardrails" fail independently. This paper introduces a framework that uses copulas to model the actual correlations between safeguard decisions, resulting in risk assessments that are far more accurate and system designs that are up to 39.6% more cost-effective while actually meeting safety requirements.

What Problem Does This Paper Address?

When we build AI agents, we don't just use one safeguard; we stack them. We might have a prompt-injection detector, a goal-alignment checker, and a final content filter. Standard practice treats these as independent components.

However, safeguards are rarely independent. They often share the same training data, use similar architectures (like transformer-based LLMs), or are defeated by the same "intrinsically difficult" attacks. If you assume two safeguards are independent, you’ll vastly underestimate the probability that they both miss a malicious action. This "independence fallacy" leads to a false sense of security and inefficient resource allocation—either spending too much on redundant checks or, worse, deploying a system that violates its own safety limits.

Methods

The authors propose a dependence-aware framework for designing and assessing multi-component safeguard systems. Their approach centers on three pillars:

  1. Copula Modeling: Instead of simple probability products, they use copulas (Gaussian, Student-t, Clayton, and Gumbel) to represent the joint distribution of safeguard decisions. This allows them to model how "misses" and "false blocks" correlate across different components.
  2. Policy Optimization (DAPOA): They developed the Dependence-Aware Policy Optimization Algorithms. These algorithms search through two main policy classes:
  3. Sequential Layered Screening: Safeguards are checked one by one, stopping immediately if any one returns a "BLOCK."
    • Selective Verification: A two-stage process where a "first stage" of safeguards routes actions to either an automatic "ALLOW," an automatic "BLOCK," or a more expensive "REVIEWER" (a high-quality model or human).
  4. Optimal Routing Theory: They prove that for selective verification, the optimal routing map follows a two-threshold rule based on the likelihood ratio of the first-stage results. Only "ambiguous" patterns—those where the first stage isn't quite sure—justify the cost of the expensive reviewer.

Datasets

The researchers evaluated their framework using:

  • $\tau^2$-Bench: A benchmark for conversational agents in retail and airline environments. They used 2,524 paired actions (one authorized, one unauthorized).
  • LLM Safeguards: They tested four specific models as safeguards: Qwen3 8B, Gemma4 E4B, Ministral 3 8B, and Claude Sonnet 5 (acting as the high-cost reviewer).

Key Insights and Ideas

  • The Cost of Ignorance: Ignoring dependence isn't just a statistical error; it's a security risk. In numerical experiments, policies designed under the assumption of independence violated security requirements in 6 out of 8 scenarios when tested against actual dependent distributions.
  • Dependence as a Design Lever: The paper shows that stronger dependence in layered screening actually increases security misses and invocation costs but decreases false blocks. Knowing this allows designers to tune the system's "aggressiveness" based on the known correlation of their tools.
  • Quality-Adjusted Review: A reviewer is only worth the cost if it provides a "quality-adjusted" improvement over the first stage. The authors provide an exact mathematical condition (Equation 3.11 in the paper) to determine if a reviewer should even be included in the policy.

Key Results

  • Massive Cost Savings: In their empirical study, the dependence-aware selection lowered the total operating cost by 39.6% compared to the independence-based selection while maintaining the same safety levels.
  • Safety Compliance: While independence-based models often "cheated" by selecting cheaper, less reliable configurations that appeared safe, the dependence-aware model correctly identified when more expensive safeguards (like Claude Sonnet) were necessary to meet strict miss-rate limits (e.g., < 0.5%).
  • Software Release: The authors released DAPOA as an open-source package, making it easier for engineers to implement these optimization algorithms in production AI pipelines.

Limitations and Future Work

  • Static Dependence: The current framework assumes the dependence structure between safeguards is fixed. In reality, an adaptive attacker might specifically craft inputs to maximize the correlation of failures across safeguards.
  • Cost Estimation: The model relies on accurate estimates for the "cost" of a security miss vs. a false block. In the real world, quantifying the cost of a security breach is notoriously difficult.
  • Live Deployment: The study was conducted on benchmark data. Future work is needed to validate how these dependence models hold up under the shifting distributions of live production traffic.

Summary updated

Abstract

Large language model (LLM) agents increasingly rely on safeguards to prevent unauthorized actions while preserving legitimate service. However, existing risk assessments often focus on individual safeguard performance while overlooking dependence among safeguard decisions. This paper develops a dependence-aware framework for risk assessment and optimal design of multi-component AI safeguard systems, jointly selecting configurations and operating policies for sequential layered screening and selective verification. We establish stochastic comparisons of layered-screening error probabilities and invocation costs under lower-orthant ordering. For selective verification, we characterize exactly which policies have error probabilities and review costs that respond monotonically to stronger dependence under supermodular ordering. For a fixed first stage and reviewer, we show that the optimal routing map is a two-threshold rule in the first-stage likelihood ratio. An accompanying open-source software package, Dependence-Aware Policy Optimization Algorithms (DAPOA), facilitates the practical implementation of the proposed framework. Numerical experiments show that ignoring dependence can lead to inaccurate risk assessments and infeasible safeguard designs, with independence-based selections violating security requirements in six of eight scenarios. An empirical study using four LLM safeguards on benchmark-derived agent actions further demonstrates the practical implications of dependence modeling. Under the principal operating setting, dependence-aware selection lowers held-out total cost by 39.6% relative to independence-based selection.
BibTeX
@misc{li2026dependence,
  author = {Yijia Li and Maochao Xu and Peng Zhao},
  title = {{Dependence-Aware Risk Assessment and Optimal Design of AI-Agent Safeguard Systems}},
  year = {2026},
  eprint = {2611.00386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00386},
}
Figure