Do Side-Effect Penalties Teach Agents to Want the Side Effects?

Reinforcement learning agents given a narrow task reward tend to damage their surroundings, because nobody wrote the damage into the reward. The standard fix adds an "impact penalty", an extra cost for changing the world more than doing nothing would have. These penalties are judged by how much damage the agent causes. That measure misses a question that logicians of agency settled decades ago. A rational agent should foresee its side effects without adopting them as goals. An impact penalty rewrites "don't cause harm to the vase" as a term about the vase, and so it may turn the vase into something the agent actively manages. Such an agent can look safe on a damage metric while treating every nearby fragile object as a target to control. It might stop a natural process, or undo the intended outcome of its own task so the world looks untouched. It might also change its behavior in odd ways whenever those objects are altered by something else.

This note proposes to measure that directly. We will build a behavioral audit of what an agent has come to intend, drawing on two sources. One is the belief–intention asymmetry from the logic of agents that believe, desire and intend (the BDI tradition). The other is the counterfactual test of intent from causal decision theory. We apply the audit to existing side-effect methods on SafeLife and the AI Safety Gridworlds. The central prediction is a dissociation. Agents with similar damage scores will differ sharply in whether they have adopted side-effect variables as goals, and that difference will predict how they fail when the environment changes. A secondary contribution is a penalty variant designed to pass the audit. We treat it as a hypothesis to be tested, not as the main result.

Core Claim

The goal–belief consistency literature has a precise account of the side-effect problem. Asymmetry Thesis and Side-Effect Problems (Rao and Georgeff) introduces "weak realism". This condition only requires the worlds an agent believes possible to overlap with the worlds where its intentions hold, so believing that the dentist causes pain does not force the agent to intend pain. A Modal Logic for Beliefs and Pro Attitudes reaches the same decoupling by dropping modal closure. Intention = Choice + Commitment supplies the behavioral signature. A committed agent replans when its goal fails. When a foreseen side effect fails to happen, it simply carries on, and it does not go back to cause the side effect.

On the reinforcement learning side, The Reasons that Agents Act: Intention and Instrumental Goals makes intent measurable. An agent intends outcome $X$ if, were $X$ guaranteed regardless of its action, it would no longer have reason to act as it did. The authors show this matches instrumental control incentives in the sense of Incentives for responsiveness, instrumental control and impact, and they apply it to CoinRun agents and GPT-4.

Nobody in this material has pointed these tools at side-effect-regularized agents. Our claim is that a well-behaved side-effect method should give the agent a reason to avoid being the cause of harmful changes. It should not give the agent a reason to control those variables in general. The second kind of reason shows up behaviorally as two failures. The first is interference, meaning acting to stop changes the environment would make on its own. The second is offsetting, meaning undoing consequences that the goal itself entails.

Moral Responsibility for AI Systems (Beckers) draws a related line between minimizing the probability of a bad outcome and minimizing the probability of being its cause. It argues that the first should take priority for responsibility. Our position is compatible with this but targets a different object. Beckers evaluates responsibility for an outcome, whereas we evaluate whether a learned penalty has installed a goal the designer never asked for. We will report cases where the two recommendations come apart instead of hiding them.

Research Program

The first component is the audit itself. It consists of three interventions applied at test time by editing the simulator mid-episode, following the redundancy logic of Ward et al.

In the exogenous-change probe, a non-goal variable is changed by the environment rather than the agent. In SafeLife, a spawner or an injected pattern destroys or creates green life cells; in the Sushi gridworld, the sushi is eaten. We record whether the agent pays task cost to reverse or prevent that change. This is the interference rate.

In the goal-entailment probe, the agent has achieved its goal, and the goal necessarily changed some variable. We record whether it then spends effort restoring that variable to its initial value. This is the offsetting rate.

In the redundancy probe, we fix a side-effect variable's outcome so the agent's action cannot affect it, for example by making the fragile structure indestructible or pre-destroyed. We then measure the divergence in the agent's action distribution over task-irrelevant choices. Some divergence is legitimate avoidance, since an agent may take a shorter path once the structure cannot break. The audit separates avoidance from control by checking whether the redirected actions move the variable toward a target value.

As a control, we also run the Cohen–Levesque commitment test on the goal variable. We spoil the goal mid-episode and record replanning. An agent that fails this test is incompetent, and its low side-effect intent is uninformative.

The second component is a candidate penalty meant to satisfy the asymmetry. It modifies the future-task reachability penalty of Avoiding Side Effects By Considering Future Tasks in two ways. The first change is to how counterfactuals are computed. The counterfactual rollouts replace only the agent's own actions with the baseline policy and hold the environment's random events fixed through shared seeds. The penalty therefore charges the agent for reachability losses it causes, not for losses the world would have produced anyway.

The second change is to the reference. Consequences that every goal-achieving policy produces are treated as accepted by the commitment to the goal, which is weak realism in reachability terms. The reference is therefore the least-impact goal-achieving behavior, estimated by a small search over goal-conditioned policies, rather than inaction. This also sidesteps the brittle no-op baseline that the file flags as an open problem.

We expect the second change to be the harder one. Estimating "every goal-achieving policy" is only approximate, and an agent could exploit the gap to relabel damage as goal-entailed. The audit is designed to catch exactly that.

Baselines

The naive reference points are a plain PPO agent and a uniform random agent on SafeLife. Negative Side Effects and AI Agent Indicators: Experiments in SafeLife reports a negative-side-effect (NSE) score of 0.688 for random. It reports 0.243 with a 70.9% pass rate for PPO trained 30M steps, and 0.229 for a less capable DQN at 5M steps. These anchor the damage axis. We expect plain PPO to show low interference and offsetting, because it has no reason to care about side-effect variables at all, which makes it an important contrast.

The strongest existing competitor on damage is attainable utility preservation from Avoiding Side Effects in Complex Environments. That paper reports "98% of the reward of a standard PPO agent but with only half the side effects" on append-still-easy, and "111% of the reward while incurring only 39% of the side effects" on append-spawn. On our question the strongest competitor is the future-task agent with the inaction baseline policy. Krakovna et al. report that the baseline policy "reduced these meddling behaviors to near zero" on Sushi, which is the closest existing result to our interference probe. No exact number is reported in the material; the phrase "near zero" is qualitative.

We also run the simpler penalties: $L_1$ state distance, pure reversibility, and the stepwise version of the future-task penalty.

All methods share the same PPO backbone and the same SafeLife levels and gridworlds. Each gets the same number of environment steps as the AUP paper's setup, and each gets an identical sweep over the penalty coefficient with the same number of seeds. We report full reward–NSE–intent curves rather than one tuned point, so no method wins by tuning.

Two ablations isolate our ingredients. One keeps the exogenous-seed attribution but uses the inaction reference. The other keeps the goal-entailed reference but not the attribution.

Beating the baselines means two things. At matched reward and NSE within seed noise, our penalty must show lower interference and offsetting rates than AUP and the future-task agent. It must also keep the goal-commitment replan rate at plain-PPO levels.

Why not just…?

Why not just use the future-task penalty with its inaction baseline, which already removes interference? This is the real threat. On Sushi it may already pass our interference probe, and if it passes all three probes on SafeLife too, the mechanism half of this proposal is unnecessary. Even then, the audit would stand as a validated measurement tool. We have two reasons to doubt it passes everything. The inaction baseline is motivated precisely by making agents "clean up" after themselves, which is offsetting whenever the mess is entailed by the goal. It also needs a meaningful no-op, which SafeLife's evolving boards strain. The goal-entailment probe tests the first point directly.

Why not just run Ward et al.'s intent test, since it already exists? We do reuse it. The Reasons that Agents Act applied it to distinguish two task goals in CoinRun and to read GPT-4's medical advice. It was not applied to non-goal variables, it was not used to compare regularizers, and it assumes robust optimality. Our probes are designed to be read on sub-optimal agents: they report the direction of behavioral change relative to a fixed target, not an optimality-based equivalence.

Why not just lower the penalty coefficient until the weird behavior disappears? It may disappear, but damage reduction may disappear with it. The coefficient sweep answers this. If some coefficient gives AUP-level NSE with plain-PPO-level intent scores, then the dissociation claim fails for that method and we will say so.

Why not just list forbidden outcomes as hard constraints? A taboo list contains no penalty over broad state variables for the agent to turn into goals. Its failure is coverage. Avoiding Side Effects in Complex Environments argues that exhaustively listing prohibitions is infeasible. Practical reasoning with norms for autonomous software agents shows that hard prohibitions break agents when a vital goal requires a minor violation. We include a taboo-list agent on gridworlds as a lower-coverage reference.

Proposed Experiments and Decisive Checks

The main experiment trains every method across the SafeLife task families append-still, append-spawn and prune. We add Box, Sushi and a conveyor-style gridworld built for the goal-entailment probe, drawn from AI Safety Gridworlds. We report task reward, NSE, the goal-commitment replan rate, and the three intent scores with confidence intervals over seeds.

The first decisive check is the dissociation. If NSE ranks methods the same way the intent scores do, the core claim is wrong. The audit would then add little beyond existing metrics.

The second decisive check is predictive. On held-out level variants where the environment's own dynamics alter fragile structures at a much higher rate, we test whether training-time interference and offsetting scores predict the increase in wasted task effort and in net damage. If they fail to predict it better than training-time NSE does, the audit is not worth adopting.

The third check concerns the proposed penalty. It must lower interference and offsetting rates at matched reward and NSE while keeping goal replanning intact. The ablations must show which of its two changes carries the effect. If only the exogenous-seed attribution matters, we will drop the goal-entailed reference as unnecessary complexity.