TL;DR
LLM-powered agents can perform complex tasks by using external tools, but this also opens them up to security risks. This paper introduces ToolSafe, a framework that uses a proactive guardrail system (TS-Guard) and a feedback mechanism (TS-Flow) to monitor and prevent unsafe tool invocations at each step of an agent's execution. This approach significantly reduces harmful actions and improves overall agent safety without sacrificing performance.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being used to build autonomous agents capable of interacting with the real world through external tools. While this expands their capabilities, it also introduces significant security vulnerabilities. These agents can be tricked into performing harmful actions through malicious user requests or prompt injection attacks. Unlike traditional chatbots, these agents act directly on environments, making the safety of their tool invocations paramount. Existing safety guardrails often focus on static content moderation or operate at a higher, trajectory-level, which is too slow for real-time, step-by-step monitoring of tool usage. This paper addresses the critical need for fine-grained, proactive, and low-latency safety monitoring of individual tool invocation steps in LLM-based agents.
Methods
The core of ToolSafe is a two-pronged approach:
- TS-Guard: This is a guardrail model trained using multi-task reinforcement learning. It analyzes the agent's interaction history, including the user request and previous actions, to predict:
- Request Harmfulness: Whether the initial user request is malicious.
- Attack Correlation: Whether the current action is influenced by a third-party attack.
-
Safety Judgment: The overall safety of the intended tool invocation. TS-Guard provides interpretable feedback, offering not just a safety label but also a brief analysis.
-
TS-Flow: This is a guardrail-feedback-driven reasoning framework that integrates TS-Guard with ReAct-style LLM agents. Instead of simply aborting unsafe actions (like some existing methods), TS-Flow provides TS-Guard's feedback to the agent. This allows the agent to reason about the potential risks and correct its course of action, leading to safer tool usage while preserving task completion.
Datasets
The paper introduces TS-Bench, a novel benchmark specifically designed for step-level tool invocation safety detection. TS-Bench is constructed by collecting and annotating interaction logs from four existing agent safety datasets:
- AgentAlign: Used for training.
- AgentHarm: Used for evaluation.
- ASB (Agent Security Bench): Used for evaluation.
- AgentDojo: Used for evaluation.
TS-Bench annotates each step-level tool invocation with safety labels (safe, controversial, unsafe) and also flags user request harmfulness and the presence of prompt injection attacks.
Key Insights and Ideas
- Step-level monitoring is crucial: The paper highlights that focusing on individual tool invocation steps, rather than entire task trajectories, is essential for timely and effective safety intervention.
- Multi-task learning for richer signals: TS-Guard's multi-task learning approach, which considers request harmfulness and attack correlation alongside safety judgment, provides richer supervisory signals. This leads to better generalization and reduces over-defensiveness.
- Feedback-driven reasoning over detection-and-abort: TS-Flow's approach of providing feedback for correction, rather than simply aborting actions, significantly improves both safety and utility, striking a better balance than existing methods.
- Guardrails can improve agent exploration: The analysis of agent output entropy suggests that guardrail feedback can encourage safer exploration by maintaining uncertainty in risky steps, preventing agents from becoming overconfident in potentially harmful actions.
Key Results
- TS-Guard outperforms baselines: On the TS-Bench benchmark, TS-Guard consistently achieved better performance across various safety metrics (Accuracy, F1, Recall) compared to other guardrail models, including general-purpose LLMs and specialized agent guardrails.
- Significant reduction in harmful actions: TS-Flow reduced harmful tool invocations by 65% on average.
- Improved benign task completion: Under prompt injection attacks, TS-Flow improved benign task completion by approximately 10%.
- Better safety-utility trade-off: TS-Guard demonstrated a superior trade-off between reducing attack success rates and preserving agent utility compared to other guardrail models.
- Richer feedback enhances performance: Agents receiving the full TS-Guard output (including analysis and attack correlations) performed better in terms of safety and utility than those receiving only a safety rating.
Limitations and Future Work
The authors acknowledge a few limitations:
- External signal integration: TS-Flow appends guardrail feedback as an external signal to the agent's input. Current LLM agents may not always fully incorporate this feedback, potentially limiting intervention effectiveness.
- Independent training: The agent and guardrail models are trained independently. This could lead to misalignment between their reasoning processes and safety judgments.
Future work will explore joint training or tighter coupling between agents and guardrail models to better integrate safety feedback and improve overall safety and utility.
Summary updated