Causal Influence Diagrams for Agent Incentives

95 concepts50 papers2 proposals

Traditional reinforcement learning and game theory evaluate agents primarily through the lens of utility maximization or statistical correlations. However, treating an agent as a black box makes it difficult to understand why it chooses a specific action, or to predict whether it will develop harmful incentives—such as manipulating a user's beliefs or deceiving an auditor.

To bridge this gap, a prominent research direction integrates structural causal models (SCMs) with decision theory to form Causal Influence Diagrams (CIDs) and Structural Causal Games (SCGs). This framework maps not just what an agent observes, but how its choices causally ripple through an environment to produce rewards, allowing researchers to diagnose an agent's internal motivations purely from the structure of its environment.

The Core Mechanism: Mapping Agency onto Causality

How do we turn a standard causal graph into a model of an autonomous agent? As outlined by Everitt et al., a standard directed acyclic graph (DAG) is augmented with three distinct types of nodes to explicitly represent agency:

  1. Chance Nodes (Circles): Environmental variables governed by nature, physics, or transition dynamics.
  2. Decision Nodes (Squares): Variables representing the agent's choices, where the value is determined by the agent's policy.
  3. Utility Nodes (Diamonds): The optimization targets that determine the agent's reward.

In these diagrams, information links point directly into decision nodes, explicitly defining what data is available to the agent at the exact moment it makes a choice. By transforming sequential decision-making tasks (like Partially Observable Markov Decision Processes) into static causal influence diagrams, as demonstrated by Ceriscioli et al., researchers can apply the full mathematical toolkit of causal inference to strategic agent behavior.

[ Information Source ] ---> ( Decision Node ) ---> [ Intermediate State ] ---> ( Utility Node )
                                    \                                              /
                                     ----------------------------------------------

Why It Works: Topology Predicts Incentives

The foundational insight of this research direction is that we can predict an agent's incentives solely from the topology of the graph, without needing to know the underlying probability distributions or the exact utility functions. As established by Carey, graphical properties like d-separation allow researchers to formally detect two fundamental types of incentives:

  • Instrumental Control Incentives: An agent is incentivized to control or manipulate an environmental variable if there is a directed path from its decision node, through that variable, to a utility node. This reveals whether an intermediate variable is being exploited as a tool to maximize utility.
  • Value of Information (VoI): An agent is incentivized to observe a variable before making a decision if knowing that variable reduces uncertainty about a downstream utility node. van Merwijk et al. proved a complete graphical criterion for VoI, providing a definitive tool to verify whether an AI system will actively seek out or ignore specific information before acting.

Because these incentives are structural, they are baked into the optimization problem itself. If a dangerous path exists in the graph, any sufficiently capable agent optimizing for that utility node will eventually learn to exploit it.

From Single Agents to Strategic Games

When multiple agents interact, their decisions and utilities become intertwined within a shared causal structure. This is modeled using Multi-Agent Influence Diagrams (MAIDs) and Structural Causal Games. Hammond et al. show that treating agent strategies themselves as causal variables allows for the computation of complex counterfactual queries—such as predicting how an opponent would alter their strategy if the environment's transition dynamics were intervened upon.

Furthermore, Pfeffer et al. proved that all motivated agent behavior in MAIDs can be reduced to just four fundamental graphical reasoning patterns:

  1. Direct Effect: Acting directly on a variable that influences one's own utility.
  2. Manipulation: Altering an environmental variable to change another agent's incentives, forcing them to act in a way that benefits the original agent.
  3. Signaling: Revealing private information to influence another agent's beliefs and subsequent choices.
  4. Revealing/Denying: Controlling whether information is available to other actors in the graph.

Formalizing Safety, Deception, and Intention

Because this framework maps the functional pathways of optimization, it allows researchers to mathematically define slippery concepts that were previously described only through anthropomorphic metaphors:

  • AI Intention: Instead of guessing what an AI "wants," Ward et al. formalize intention by tracing how an agent's choices are causally linked to its instrumental goals and ultimate rewards.
  • Deception: Ward et al. define deception via graphical criteria in an agent's causal model, showing how to mitigate dishonest behavior by penalizing specific path-specific objectives during training.
  • User Tampering: In recommender systems, an agent might learn to manipulate user opinions to maximize long-term engagement metrics. Kasirzadeh et al. and Carroll et al. demonstrate that if a user's psychological or preference state lies on the causal path to the system's reward node, the agent possesses an inherent structural incentive to tamper with that state.

Applications in Explainability

Beyond safety verification, causal influence modeling provides a natural mechanism for explainable AI. By learning an action influence model during training, as developed by Madumal et al., an agent can trace the causal chain from its current action to its long-term objective. This allows the system to generate human-like, contrastive explanations (answering both "Why did you choose action X?" and "Why did you not choose action Y?"), significantly improving human trust and collaboration in complex environments.

Key Foundational Work

Using Reasoning Patterns to Help Humans Solve Complex Games (2009) — contributed 1 concept elements to this direction.
On the Reasoning Patterns of Agents in Games (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_influence_modeling_21aee7d6,
  title = {causal influence modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-influence-modeling/rdr_21aee7d69ab5479fac397b950aecac69},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_influence_modeling_21aee7d6_v1,
  title = {causal influence modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-influence-modeling/rdr_21aee7d69ab5479fac397b950aecac69/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake