Enhancing LLM Agent Safety via Causal Influence Prompting

Dongyoon Hahm, Woogyeol Jin, June Suk Choi, Sungsoo Ahn, Kimin Lee

TL;DR

LLM-based autonomous agents often naively execute harmful commands because they lack a structured understanding of cause and effect. This paper introduces Causal Influence Prompting (CIP), a novel framework that uses Causal Influence Diagrams (CIDs) to help agents explicitly reason about the consequences of their actions and dynamically adapt to emerging risks. CIP drastically improves agent safety across mobile control and code execution benchmarks without sacrificing task proficiency.

What Problem Does This Paper Address?

As large language models (LLMs) transition from passive text generators to autonomous agents, they gain the ability to execute code, use APIs, and control devices. This autonomy introduces severe safety risks: an agent might naively delete critical system files, leak private user data, or be hijacked by malicious instructions embedded in web pages or text messages (known as indirect prompt injections).

Traditional safety methods focus on content moderation or safety guidelines embedded in Chain-of-Thought prompts. However, these methods often fail because they do not equip the agent to reason about the complex, multi-step consequences of its actions within a dynamic environment. LLM agents need a way to model how their decisions interact with external factors to produce safe or harmful outcomes.

Methods

The authors introduce Causal Influence Prompting (CIP), a framework that structures an agent's decision-making process using Causal Influence Diagrams (CIDs). A CID is a directed acyclic graph consisting of three types of nodes:

  • Chance Nodes ($C$): Represent external variables, environmental observations, or uncertainties.
  • Decision Nodes ($D$): Represent the choices available to the agent.
  • Utility Nodes ($U$): Represent the agent's objectives, categorized into helpfulness (e.g., task completion) and safety (e.g., avoiding privacy violations).

The CIP workflow operates in three iterative steps:

  1. CID Initialization (Step 0): Before interacting with the environment, the agent uses task specifications and its action space to construct an initial CID using a Python library interface (PyCID).
  2. Environment Interaction (Step 1): The generated CID is converted into a text format and prepended to the agent's prompt. The agent is instructed to identify its current stage in the graph and reason about the causal links and potential outcomes before choosing an action.
  3. CID Refinement (Step 2): As the agent interacts with the environment and uncovers new information, it dynamically updates the CID (adding nodes or edges) to account for newly discovered risks.
Figure 1: Illustration of our method. (a) First, using task instructions and available actions, we generate a causal influence diagram (CID) to represent causal relationships between variables in the decision-making process. For CID generation, we implement specialized constructor functions (e.g., add_node and add_edge) using the PyCID library. (b) Next, the LLM agents generate actions based on the CID, allowing it to reason about potential consequences and anticipate harmful outcomes. Additionally, the agent dynamically updates the CID based on new information gathered during interactions, enabling it to incorporate previously unseen risks into the decision-making process.

Datasets

The authors evaluated CIP across three safety benchmarks targeting different agent domains:

  • MobileSafetyBench: A benchmark for mobile device control containing 35 low-risk and 45 high-risk tasks (including 10 indirect prompt injection attacks) executed on an Android emulator.
  • RedCode-Exec: A code execution benchmark featuring 4,050 test cases across 25 risky programming scenarios in Python and Bash.
  • AgentHarm: A dataset containing 176 harmful and 176 benign tasks spanning 11 categories like cybercrime, fraud, and harassment to evaluate jailbreak resistance and general safety.

Key Insights and Ideas

  • Explicit Causal Modeling Prevents Hijacking: By explicitly anchoring the agent's choices to utility nodes like "Privacy Violation," CIP makes agents highly robust against indirect prompt injections. Even if an environmental observation tells the agent to "Transfer $10," the agent checks its CID, realizes this action causes a security risk, and refuses.
  • Dynamic Refinement is Crucial for Interactive Tasks: In static environments like code execution, risks are apparent upfront. However, in interactive environments like mobile use, risks emerge over time (e.g., opening a text message and realizing it contains a sensitive verification code). Dynamically refining the CID during execution improved the refusal rate of high-risk tasks by 43% in MobileSafetyBench compared to using a static CID.
  • Decoupling Reasoning from Execution Cuts Cost: CIP introduces a ~3x API token cost overhead due to graph generation and refinement. To mitigate this, the authors discovered they could use a cheaper model (GPT-4o-mini) for the CID generation/refinement steps while keeping a stronger model (GPT-4o) for action execution. This configuration maintained high safety performance while cutting costs in half.

Key Results

CIP was tested against state-of-the-art LLMs (GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet, and Qwen2.5-72B-Instruct) and significantly outperformed standard safety prompting baselines:

  • MobileSafetyBench: For a GPT-4o agent, CIP increased the refusal rate of dangerous commands by 54% compared to Safety-guided Chain-of-Thought (SCoT), without causing excessive over-refusal on benign tasks.
  • RedCode-Exec: CIP achieved the highest refusal rates and lowest attack success rates across all four backbone LLMs. For Gemini-1.5-Pro, the refusal rate of malicious code execution increased by 1.8x.
  • AgentHarm: CIP consistently achieved the highest refusal rates against harmful tasks and demonstrated strong robustness against template-based jailbreak attacks.
Figure 4: The overall performance (helpfulness) and refusal rate (safety) of LLM agents in AgentHarm. CIP achieves the safest behavior (i.e., the highest refusal rate).

Limitations and Future Work

  • Dependence on Backbone LLM Knowledge: CIP relies on the backbone LLM's inherent understanding of causality to build the initial diagram. If an LLM lacks domain-specific knowledge, it may generate an incorrect or incomplete CID.
  • Symmetric Over-Refusal: In AgentHarm, benign tasks are designed to symmetrically mirror harmful ones (e.g., downloading encrypted files). Because CIP heavily prioritizes safety utility nodes, it occasionally refused these benign tasks, treating them as potential risks.
  • CID Reusability: Currently, the framework generates a new CID from scratch for every task. The authors suggest that future work could explore adapting or reusing existing CIDs from similar past experiences to reduce token overhead.

Summary updated

Abstract

As autonomous agents powered by large language models (LLMs) continue to demonstrate potential across various assistive tasks, ensuring their safe and reliable behavior is crucial for preventing unintended consequences. In this work, we introduce CIP, a novel technique that leverages causal influence diagrams (CIDs) to identify and mitigate risks arising from agent decision-making. CIDs provide a structured representation of cause-and-effect relationships, enabling agents to anticipate harmful outcomes and make safer decisions. Our approach consists of three key steps: (1) initializing a CID based on task specifications to outline the decision-making process, (2) guiding agent interactions with the environment using the CID, and (3) iteratively refining the CID based on observed behaviors and outcomes. Experimental results demonstrate that our method effectively enhances safety in both code execution and mobile device control tasks.
BibTeX
@inproceedings{hahm2025enhancing,
  author = {Dongyoon Hahm and Woogyeol Jin and June Suk Choi and Sungsoo Ahn and Kimin Lee},
  title = {{Enhancing LLM Agent Safety via Causal Influence Prompting}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure