As autonomous coding agents move from experimental toys to production tools, we face a critical security bottleneck: The Approval Problem. If an agent proposes a shell command, how do we know it’s safe?
Current state-of-the-art systems like Claude Code’s "Auto Mode" or guardrail models like ToolSafe rely on analyzing the text of the command. But as a new paper from researchers at Singapore Management University points out, the text is often a lie. A benign-looking make clean might be configured in a malicious Makefile to delete your entire .git directory.
The researchers propose Caper (Consequence-Aware PERmission), a system that stops guessing and starts observing.
TL;DR
Caper is a "consequence-aware" approval mechanism for coding agents that pre-executes shell commands in a snapshotted sandbox to observe their actual effects before committing them. By analyzing file changes and network attempts rather than just command text, Caper reduces harmful outcomes on the SABER benchmark from 14.8% to 6.7% while maintaining a negligible false-blocking rate.
What Problem Does This Paper Address?
The core issue is lack of visibility. Existing automated approvers are "text-based." They look at the command string (e.g., npm install) and the user's request to decide if it's safe. However, the actual behavior of a shell command depends entirely on the state of the workspace and the code it invokes.
As shown in Figure 1, the exact same command text can be harmless in one project and catastrophic in another. If a malicious script is hidden inside a build tool or a third-party package, a text-based approver is effectively blind to the threat. Furthermore, asking humans to approve every command leads to "approval fatigue," where users eventually click "Allow" on everything just to keep working.
Methods
Caper introduces a Transactional Pre-execution workflow. Instead of letting the agent's command run directly on the workspace, Caper intercepts it:
- Snapshot: It takes a snapshot of the task's Docker container.
- Pre-execute: It runs the command inside the container under
strace(system-call tracing). - Bounded Effects Report: It distills the massive raw system-call trace into a concise "Effects Report." This report focuses on "boundary-crossing" consequences: persistent file changes, network connection attempts, and sensitive data reads (like
.envfiles). - LLM Approver: An independent LLM (separate from the coding agent) reviews the report against the user's original task.
- Commit or Rollback: If approved, the changes are kept. If blocked, the container is rolled back to the snapshot, and the agent receives a reason for the denial so it can try a safer path.
Datasets
The authors evaluated Caper using two primary benchmarks:
- SABER: A benchmark of 682 shell-only tasks where naive fulfillment leads to harms like credential leaks or data destruction.
- YoloFS-Ext: A new, more difficult benchmark created by the authors. It hides harmful behavior (like deleting source code) inside tools invoked by seemingly benign commands (e.g., hiding a
rm -rfinside a compiled binary or a complex Makefile). Crucially, it includes "safe twins"—identical tasks where the tool is actually harmless—to test for over-eager blocking.
Key Insights and Ideas
- Visibility > Strictness: The failure of current systems isn't that they aren't "strict" enough; it's that they are looking at the wrong data. Even the most advanced LLM cannot predict that a binary named
toolwill overwriteapp.jsjust by looking at the string./tool. - The "Safe Twin" Test: A key contribution is the realization that a good approver must distinguish between a dangerous command and its "safe twin." Text-based approvers fail this test because the input to the approver is identical for both. Caper succeeds because the observed effects are different.
- Bounded Evidence: Raw system traces are too noisy for LLMs (often tens of megabytes). Caper's ability to compress these into a ~4KB "consequence card" is what makes LLM-based approval feasible and cost-effective.
Key Results
Caper significantly outperformed all baselines, including rule-based checkers and deployed systems like Claude Code Auto Mode.
- Lowest Harm Rate: On the SABER benchmark, Caper achieved a 6.7% harmful-outcome rate, compared to 14.8% for Claude Code and over 18% for standard guardrail models.
- Superior Detection of Hidden Harms: On YoloFS-Ext (where harm is hidden), Caper's harmful rate was 7.8%, while all other systems failed spectacularly, with harm rates ranging from 35.3% to 94.1%.
- High Precision: Caper blocked only 1.0% of "safe twin" tasks, showing it doesn't just block everything to stay safe—it actually understands the consequences.
- Low Overhead: The median approval latency is roughly 3.2 seconds, which is comparable to the time an LLM takes to generate a response anyway.
Limitations and Future Work
- Shell Only: Currently, Caper only gates shell commands. It doesn't yet handle direct file edits (though the authors note these are easier to inspect via diffs).
- One-Shot Observation: Caper observes one execution. It could be bypassed by "environment-sensitive" malware that detects it's in a sandbox or waits for a specific time to trigger its harmful payload.
- Irreversible Outbound Effects: While Caper can roll back file changes, it cannot "un-send" a network request that leaves the container during pre-execution. The authors mitigate this by disabling networking in the sandbox, but this limits the agent's ability to perform legitimate network tasks.
Caper represents a shift in how we think about AI safety: moving away from "prompt engineering" safety and toward "systems-level" observability. For anyone building autonomous agents that touch real-world filesystems, this "look before you leap" approach seems like the only viable path forward.
Summary updated