This summary explores the paper "guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems," which introduces a framework for systematically measuring the security coverage of Large Language Model (LLM) agent deployments.
TL;DR
LLM agents can execute actions through numerous paths—like spawning subprocesses or direct OS calls—that often bypass standard security gateways. guardscope is a new framework that uses kernel-level monitoring (eBPF) to verify exactly which of these actions a runtime security control actually inspects, revealing significant blind spots in current deployment strategies.
What Problem Does This Paper Address?
As enterprises deploy LLM agents to handle internal data and services, a new ecosystem of "runtime controls" (gateways, guardrails, and policy engines) has emerged to monitor their behavior. However, these controls have a fundamental limitation: they can only check actions that cross the specific point where they are placed.
For example, an "agent gateway" might perfectly inspect standard HTTP tool calls but be completely blind if the agent harness spawns a local child process or opens a raw network socket directly. Currently, there is no systematic way to measure this "placement-driven coverage" against a reliable ground truth. The authors address this gap by providing a framework to make these coverage claims verifiable.
Methods
The authors developed guardscope, a two-stage framework for measuring coverage:
- Static Analysis: This stage analyzes a runtime control’s source code and documentation to predict which action paths it can check based on its placement and parsing capabilities.
- Empirical Measurement: This stage runs a reference workload on a real deployment. The framework uses a "runner" to drive an agent harness through deterministic replays (avoiding LLM variability). Crucially, it uses Tracee (an eBPF-based sensor) to provide kernel-level ground truth. By reconciling the control's own logs against what the host kernel actually observed, guardscope classifies every action as checked, bypassed, or prevented.
To provide a theoretical foundation, the authors enumerated 15 distinct action paths (e.g., in-process tool calls, subprocess execution, direct network egress) and 8 common topology designs (ranging from bare harnesses to kernel-level enforcement).
Datasets
The paper does not use traditional ML training datasets. Instead, it utilizes:
- Reference Workload: A fixed set of 15 action instances designed to exercise key paths like HTTP/stdio tool calls, direct OS operations, and network egress.
- Agent Harnesses: Evaluated using Pi (v1.0.4) and OpenClaw (2026.9.3).
- Runtime Controls: Tested against real-world tools including Microsoft’s Agent Governance Toolkit (AGT), the open-source agentgateway, and ActPlane (a kernel-level control).
Key Insights and Ideas
- Placement is Destiny: A control's coverage is strictly limited by its position in the system architecture. A gateway in front of a harness is semantically rich (it sees tool arguments) but architecturally shallow (it is easily bypassed).
- The Coverage-Semantic Trade-off: There is an inverse relationship between how many paths a control can see and how much semantic detail it can inspect. Kernel-level controls (T7) see almost every path but lack the "agent context" (e.g., they see a file open but don't know which specific tool call triggered it).
- Composition Does Not Expand Coverage: Combining multiple "sub-kernel" controls (e.g., a gateway plus an in-harness check) adds more features to inspect, but it does not close the fundamental gaps (like direct OS operations) that only a kernel-level control can reach.
- Bypasses are Common: The study found that 5 out of the 15 identified action paths bypass every common design that sits below the operating system kernel.
Key Results
The empirical runs confirmed the authors' theoretical derivations for coverage outcomes.
- Gateway Controls (T1-T3): Microsoft's AGT, when deployed as a gateway, correctly recorded decisions for HTTP tool calls but was bypassed by all other exercised classes (file writes, process spawns, etc.).
- Kernel-Level Controls (T7): ActPlane caught six actions and denied one (a subprocess spawn), but it still missed some actions (like UDP sends) due to specific gaps in its eBPF hook implementation, highlighting that even kernel-level placement requires comprehensive hook coverage.
- Stability: Outcome counts were remarkably stable across different harnesses (Pi vs. OpenClaw) and hardware architectures (arm64 vs. x86_64), suggesting that the findings are fundamental to the topology designs themselves rather than specific implementations.
Limitations and Future Work
- Incomplete Empirical Validation: The current reference workload only exercises 4 of the 15 identified paths; the remaining 11 rely on theoretical design analysis.
- Coverage vs. Effectiveness: "Checked" only means the action crossed an enforcement point; it doesn't guarantee the security policy itself was well-configured or effective at stopping an actual attack.
- Kernel Noise and Attribution: eBPF sensors are noisy, observing many processes unrelated to the agent. Attributing a specific syscall to a specific high-level agent action remains a challenge.
- Future Directions: The authors suggest exercising conditional paths with dedicated workloads and evaluating policy effectiveness against adversarial (red-teaming) workloads.
Summary updated