A Multi-Objective Approach to Mitigate Negative Side Effects

Sandhya Saisubramanian, Ece Kamar, Shlomo ZilbersteinIJCAI 2020 · DOI

Introduction: The Challenge of Negative Side Effects

Autonomous agents are typically designed with a specific primary goal, such as moving an object to a target location as efficiently as possible. However, an agent's internal model of the environment is often incomplete, leading to unintended consequences known as Negative Side Effects (NSE). For instance, an agent tasked with moving a box might push it across a rug, inadvertently damaging it because "rug cleanliness" was not part of its original reward function.

Figure 1: Illustration of a negative side effect where an agent's policy dirties a rug while pursuing a primary goal.

Redesigning an agent's model from scratch every time a new side effect is discovered is impractical. This paper introduces a framework that allows agents to incorporate feedback about NSEs and adjust their behavior without compromising their primary mission.

Methodology

The proposed framework operates in three distinct phases to mitigate side effects while maintaining task performance:

Figure 2: The three-step framework: data collection via feedback, learning the NSE penalty model (RN), and computing the optimal policy using multi-objective planning.
  1. Data Collection: The agent gathers information regarding NSE penalties through human feedback (e.g., corrections or approvals) or autonomous exploration.
  2. Model Learning: Using the collected data, the agent trains a Random Forest regression model ($R_N$) to predict NSE penalties for state-action pairs, effectively augmenting its original world model.
  3. Multi-Objective Planning: The task is reformulated as a Lexicographic Markov Decision Process (LMDP). Objectives are ranked, with the primary task prioritized over NSE minimization. A key innovation is the introduction of slack ($\delta$), which defines the maximum allowable sacrifice in primary reward to reduce side effects.

Experimental Evaluation

The framework was evaluated in two simulated environments:

  • Boxpushing: An agent must move a box to a goal, where pushing it over specific surfaces incurs varying NSE penalties.
  • Driving: An autonomous driving task where the agent must reach a destination while avoiding splashing pedestrians by driving through puddles at high speeds.

Key Results

  • Effective Mitigation: The framework significantly reduced NSE penalties across both domains as the feedback budget increased.
  • Feedback Efficiency: Using "Strict Approval" feedback (HA-S), agents achieved near-optimal safety with as few as 500 samples in complex environments.
  • The Role of Slack: Experiments demonstrated that allowing a small amount of slack (allowing the agent to be ~15% less efficient) often led to a dramatic decrease in side effects, highlighting that strict efficiency often conflicts with safety.

Discussion and Insights

The authors argue that a lexicographic approach is superior to standard scalarization (combining rewards into a single weighted sum) because it avoids the difficulty of tuning weights between disparate units (e.g., time vs. dirtiness). Furthermore, the framework is robust enough to handle unavoidable NSEs by using the slack budget to minimize damage rather than failing the task.

Limitations and Future Work

Current limitations include the assumption that side effects correlate with existing state features and that the environment remains stationary. Future research will address "latent" or delayed side effects that are not immediately observable.

Summary updated

BibTeX
@inproceedings{saisubramanian2020multi,
  author = {Sandhya Saisubramanian and Ece Kamar and Shlomo Zilberstein},
  title = {{A Multi-Objective Approach to Mitigate Negative Side Effects}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/50},
}
Figure