Separating Intended Goals from Foreseen Consequences

39 concepts19 papers3 proposals

Imagine an autonomous delivery drone tasked with flying a critical medical supply to a hospital. The drone's planning algorithm determines that the fastest route requires flying through a narrow residential alley. The drone also foresees an inevitable side effect: its loud rotors will wake up a sleeping infant in one of the houses.

The drone does not want to wake the baby; its sole objective is to deliver the medicine. If it could magically silence its rotors, it would gladly do so. Yet, if the drone flies down the alley, the baby will wake up. Did the drone intend to wake the baby?

To a human observer, the answer is clearly no. Waking the baby is a foreseen but undesired side effect. However, early formal frameworks in artificial intelligence struggled deeply with this distinction, a dilemma known as the side-effect problem. If an AI cannot distinguish between what it purposefully aims to achieve and the collateral damage it merely expects to happen, it cannot make rational trade-offs, accurately interpret human motives, or be held accountable for its actions.

The Trap of Logical Closure

Why did early computational models fail at this seemingly simple distinction? The root of the problem lies in how classical logics handle an agent's mental states.

In traditional Belief-Desire-Intention (BDI) frameworks, an agent's knowledge is often modeled using normal modal logics and Kripke semantics. These systems inherently operate under logical closure: if an agent intends a proposition $P$, and believes that $P$ logically or causally implies $Q$, the mathematics forces the agent to also intend $Q$.

As Chen et al. point out, this assumption of "logical omniscience" creates highly unrealistic models of intelligence. Under standard logical closure, our delivery drone is mathematically required to adopt "waking the baby" as an active intention. This conflation makes it impossible for an agent to isolate its specific commitments from its general knowledge of causal consequences.

Solution 1: Intentions as Irreducible Primitives

The first major step toward solving this problem was to change the structural status of intention. If intentions are merely derived as a byproduct of an agent's beliefs and goals, the logical leakage of side effects is inevitable.

To break this coupling, Rao et al. introduced a semantic constraint known as "weak realism." Instead of defining intentions through other mental states, the authors treated intentions as irreducible, independent modal operators. By giving intentions their own distinct structural boundaries, they ensured that an agent could believe a side effect would occur without being logically compelled to adopt it as a personal objective.

Solution 2: The Commitment Tracking Test

While making intentions primitive fixes the logic, it raises a practical question: how can an external observer—or the agent itself—test whether an outcome was truly intended or merely foreseen?

Cohen et al. provided a foundational answer by defining intention as a combination of choice and persistent commitment. The core intuition is behavioral: how does the agent respond to failure?

               [ Action Performed ]
                       |
            +----------+----------+
            |                     |
     [ Primary Goal ]      [ Side Effect ]
         Fails?                Fails?
            |                     |
    (Agent Re-plans &     (Agent Ignores &
     Tries Again)          Proceeds Normally)

If our delivery drone flies down the alley but the baby happens to be wearing earplugs and does not wake up, a rational drone will not turn around, land, and honk its horn to finish the job. It simply proceeds to the hospital. Because the drone does not track and attempt to recover the missing side effect, commitment-based filtering classifies the outcome as a foreseen consequence rather than a formal intention.

Solution 3: Altering the Underlying Logic

Other researchers realized that to fully eliminate the side-effect problem, the rules of the logic itself had to change.

Konolige et al. proposed a "representationalist" theory of intention. Instead of merging all of an agent's goals into a single conjunctive pool, they modeled separate intentions as distinct cognitive structures. They defined these intentions by looking at "unacceptable worlds"—the specific states of affairs an agent actively rejects. This structural separation allows an agent to commit to a plan without being committed to its collateral damage, giving it the logical basis to prefer an alternative plan if one becomes available.

Taking a different mathematical route, Su et al. replaced standard Kripke structures with "linear neighborhood frames" in a non-normal modal logic. Because non-normal modal logics do not automatically close under logical consequence, they successfully decoupled an agent's pro-attitudes (like desires and goals) from their prerequisites or side effects.

Why It Matters: Social Causality and Blame

Resolving the side-effect problem is not just an exercise in pure logic; it is a prerequisite for building AI systems that can operate safely in human society.

Standard AI models typically focus on physical cause-and-effect. However, Gratch et al. argue that physical causality is entirely insufficient for interpreting human motives or resolving social conflicts. To interact naturally with humans, machines must model social causality, which incorporates epistemic variables like intention, foreknowledge, and freedom of choice.

When a security protocol fails or an autonomous vehicle causes an accident, technical causal attribution can tell us what happened, but it cannot assign moral or legal blame. As Chockler et al. and Datta et al. note, formal frameworks that lack a mechanism to explicitly model internal intentions will mistakenly assign identical culpability scores to accidental outcomes and malicious actions.

To bridge this gap, modern frameworks embed intentionality directly into structural causal models. For example, Halpern et al. use utility theory and counterfactuals to evaluate whether a harmful outcome was a primary driver of an action or a merely accepted side effect. To refine this, Beckers introduced Counterfactual NESS (CNESS), a hybrid epistemic metric that aligns automated blame attribution with human moral intuition. By grounding intention in causality, recent work by Ward et al. ensures that agents are only credited with intending outcomes they actually had the power to change, avoiding errors where agents are blamed for inevitable environmental factors.

Ultimately, by separating what an agent explicitly sets out to do from what it merely foresees, these frameworks allow autonomous systems to perform complex means-end reasoning while remaining bound to human-like standards of accountability.

Key Foundational Work

Asymmetry Thesis and Side-Effect Problems in Linear-Time and Branching-Time Intention Logics (1991) — contributed 7 concept elements to this direction.
Asymmetry Thesis and Side-Effect Problems in Linear-Time and Branching-Time Intention Logics (1991) — contributed 6 concept elements to this direction.
Intention = Choice + Commitment (1987) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 17 papers
  2. Version 1 · 2026-10-04 · 20 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_side_effect_problem_15b6a341,
  title = {side-effect problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/side-effect-problem/rdr_15b6a3416f2d434b8c9fba38febea014},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_side_effect_problem_15b6a341_v2,
  title = {side-effect problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/side-effect-problem/rdr_15b6a3416f2d434b8c9fba38febea014/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake