If an agent intends to cure a patient by administering a drug, and they know the drug causes nausea, do they intend for the patient to be nauseous?
To a human, the answer is clearly no. The nausea is a foreseen but unintended side effect. However, for decades, the formal logics used to build AI agents struggled with this distinction. In standard "Normal Modal Logics," if an agent intends $A$, and they believe $A$ logically or inevitably leads to $B$, the math forces them to intend $B$ as well. This is known as the Side-Effect Problem (or the problem of consequential closure).
Solving this problem is not just a philosophical exercise; it is a technical requirement for building agents that can be held responsible for their actions. If an AI cannot distinguish between its primary goals and the collateral damage it merely expects, it cannot make rational trade-offs or be accurately judged for its "moral" choices.
The Mechanism of Persistent Commitment
The foundation for solving this problem lies in how we define "intention." In their seminal work, Intention = Choice + Commitment, Philip R. Cohen and Hector J. Levesque argued that intention is more than just a desire; it is a persistent commitment.
The intuition here is the "tracking" test:
- Goal: If an agent intends to reach a destination and the road is blocked, they will look for a detour. They track the outcome and take corrective action if it fails.
- Side Effect: If an agent intends to drive to the destination and knows it will wear down their tires, they do not "track" the tire wear. If the tires don't wear down (perhaps the road was smoother than expected), the agent does not pull over and scrape them to ensure the wear happens.
By defining intention through this lens of commitment, Cohen and Levesque provided a way to filter out side effects. An agent is only formally committed to the outcomes they would actively try to recover if they were thwarted.
Breaking the Logic of Consequential Closure
Even with the concept of commitment, the underlying mathematics of agent logic often remained "too strong." Kurt Konolige and Martha E. Pollack pointed out in A Representationalist Theory of Intention that standard tools like Kripke structures are inherently prone to the side-effect problem because they treat all "possible worlds" equally.
To fix this, researchers moved toward Non-Normal Modal Logics. For instance, Kaile Su and colleagues introduced Linear Neighborhood Frames to model "pro-attitudes" like goals and desires. Unlike standard logic, these frames do not automatically close under logical consequence. This allows an agent to desire a specific state (e.g., "being cured") without being mathematically compelled to desire every prerequisite or implication (e.g., "paying a high medical bill").
Similarly, Anand S. Rao and Michael P. Georgeff proposed "weak realism" in BDI (Belief-Desire-Intention) logics. By requiring only a partial overlap between what an agent believes will happen and what they intend to happen, they created a formal "asymmetry" that protects agents from being blamed for every foreseen consequence.
Quantifying Responsibility and Blame
Once we can distinguish intention from side effects, we can begin to quantify blame. Joseph Halpern and Hana Chockler pioneered the use of structural causal models to move beyond binary "yes/no" causality.
In Responsibility and Blame: A Structural-Model Approach, they provide a framework to measure how much an agent contributed to an outcome. This is particularly useful in multi-agent systems where outcomes are "overdetermined." For example, in an 11–0 landslide vote, no single voter is pivotal, but each still bears some degree of responsibility.
However, causal responsibility is not the same as moral blameworthiness. To bridge this gap, Halpern and Max Kleiman-Weiner integrated utility theory into these causal models in Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility. They argue that blameworthiness depends on:
- The Causal Path: Did the agent's action actually cause the result?
- The Intentional Path: Was the result a primary driver of the agent's choice, or a side effect?
- The Counterfactual Path: Could the agent have chosen a different action with a better expected outcome?
The Epistemic Condition
The final piece of the puzzle is knowledge. An agent cannot be blamed for a side effect if they had no way of knowing it would happen, or if they didn't know they had a strategy to prevent it.
Pavel Naumov has formalized this "epistemic condition" for blameworthiness. In Knowledge and Blameworthiness, he argues that a coalition of agents is only blamable if they knew they had a strategy to prevent an outcome and knew how to execute it. This prevents "strict liability" in AI systems, where an agent might be penalized for an outcome that was technically preventable but practically unforeseeable given their limited information.
More recently, Sander Beckers refined these ideas for AI specifically, proposing the Counterfactual NESS (CNESS) metric. This approach aligns formal logic with human moral intuition by evaluating whether an agent's intent and knowledge justify the "praise" or "blame" assigned to their autonomous decisions.