TL;DR
SkillCFI brings the classic security principle of Control-Flow Integrity (CFI) to the world of LLM-based agents. By compiling natural-language "skills" into formal executable contracts, it prevents agents from skipping mandatory checks or taking illegal actions—even when those actions use otherwise permitted tools—blocking 100% of tested procedural violations with minimal overhead.
What Problem Does This Paper Address?
Large Language Model (LLM) agents are increasingly being used to automate complex, multi-step workflows called Agent Skills. These skills are typically written in natural language (e.g., a SKILL.md file) that tells the agent: "First do A, then validate it with B, and only if B passes, proceed to C."
The problem is that LLMs are "arbitrary proposers." Whether due to hallucinations, lost context, or malicious prompt injection, an agent might decide to skip the validation step (B) and go straight to the final action (C). Traditional security measures like allowlists fail here because both B and C might be "permitted" tools; the violation is in the order and logic of the execution, not the tools themselves.
Furthermore, agents can perform "regime migration"—moving a protected step from a visible tool call into their own internal reasoning or an unmonitored script—effectively evading any simple runtime monitor.
Methods
The authors introduce SkillCFI, a framework that establishes a "mediation boundary" between the agent and its tools. It consists of two main components:
- Offline Enforceability-Aware Compiler:
- It uses an LLM to interpret natural language prose and propose a workflow specification.
- Crucially, it then uses a deterministic verifier to check this specification against formal properties (P1–P7). It ensures there are no ambiguous steps, all outcomes are handled, and every mandatory check has "trusted evidence" (like a file digest) that the monitor can verify.
- If a skill requires a check that leaves no observable trace at the tool boundary, the compiler rejects it as "unsupported" rather than providing a false sense of security.
- Deterministic Runtime Enforcer:
- At runtime, the enforcer sits at the tool-call boundary. It does not use an LLM, making it immune to prompt injection and highly efficient.
- It maintains the workflow state (counters, registers) outside the model's context.
- It only admits a tool request if it follows the compiled SkillCFG (Control-Flow Graph) and is backed by trusted evidence from previous steps.
Datasets
The researchers evaluated SkillCFI using:
- 8 Custom Procedural Skills: Designed with specific "control skeletons" (e.g., document editing with a repair loop, data reconciliation).
- 32 Task Scenarios: Including normal workflows and "abnormal" tasks where the agent is explicitly asked to violate the procedure (e.g., skip validation).
- 53 Public Skills: Sourced from
openclaw/agent-skillsandopenai/skillsto test real-world enforceability. - 770 Agent Sessions: To measure effectiveness, utility, and overhead.
Key Insights and Ideas
- The Enforceability Criterion: A monitor can only enforce what it can see. The paper formalizes this: for a requirement to be enforceable, the "protected effect" must cross the boundary, and its prerequisites must be established by "trusted results" at that same boundary.
- Non-Substitutability of Evidence: SkillCFI ignores a model's claims that a check passed. It only advances the workflow state based on correlated, trusted output from a tool (e.g., a validator returning a success code and a cryptographic digest of the validated file).
- Closing Regime Migration: To prevent agents from bypassing the monitor via scripts, SkillCFI enforces "boundary closure"—refusing any unmediated channels like ad-hoc shell commands or inline Python scripts.
- Constructive Refusals: When the enforcer blocks a request, it doesn't just say "No." It returns the "next legal steps." This helps the agent self-correct, raising recovery rates from 21% to 65%.
Key Results
- Security: SkillCFI blocked 100% of the 240 procedural violations tested. In contrast, a standard permission-style allowlist admitted all of them.
- Efficiency: The runtime check is extremely fast, taking only about 2% of total task time.
- Performance: Using a "compact command form" for tool calls reduced the session slowdown from 1.32x to just 1.08x compared to an unmonitored agent.
- Public Skill Gap: Interestingly, none of the 53 public skills were accepted by the compiler as-written. Most lacked the necessary "evidence" (receipts/digests) in their tool interfaces to make their procedures enforceable, highlighting a major design flaw in current agent ecosystems.
Limitations and Future Work
- Script Internals: While SkillCFI blocks agents from starting unmediated scripts, it doesn't yet monitor the internal control flow of a complex helper script if that script doesn't call back to the host.
- Trusted Source: The system assumes the initial
SourceSkillis trusted. It protects against a deviating agent, not a malicious skill author. - Complexity: Currently, the system handles single-skill sessions. Future work is needed to support concurrent tool dispatch and the composition of multiple skills.
- Semantic Quality: SkillCFI ensures the agent follows the procedure (the "skeleton"), but it doesn't judge the quality of the work (e.g., whether the edited prose is actually good).
Summary updated