Introduction: The Challenge of Negative Side Effects
Autonomous agents are typically designed with a specific primary goal, such as moving an object to a target location as efficiently as possible. However, an agent's internal model of the environment is often incomplete, leading to unintended consequences known as Negative Side Effects (NSE). For instance, an agent tasked with moving a box might push it across a rug, inadvertently damaging it because "rug cleanliness" was not part of its original reward function.
Redesigning an agent's model from scratch every time a new side effect is discovered is impractical. This paper introduces a framework that allows agents to incorporate feedback about NSEs and adjust their behavior without compromising their primary mission.
Methodology
The proposed framework operates in three distinct phases to mitigate side effects while maintaining task performance:
- Data Collection: The agent gathers information regarding NSE penalties through human feedback (e.g., corrections or approvals) or autonomous exploration.
- Model Learning: Using the collected data, the agent trains a Random Forest regression model ($R_N$) to predict NSE penalties for state-action pairs, effectively augmenting its original world model.
- Multi-Objective Planning: The task is reformulated as a Lexicographic Markov Decision Process (LMDP). Objectives are ranked, with the primary task prioritized over NSE minimization. A key innovation is the introduction of slack ($\delta$), which defines the maximum allowable sacrifice in primary reward to reduce side effects.
Experimental Evaluation
The framework was evaluated in two simulated environments:
- Boxpushing: An agent must move a box to a goal, where pushing it over specific surfaces incurs varying NSE penalties.
- Driving: An autonomous driving task where the agent must reach a destination while avoiding splashing pedestrians by driving through puddles at high speeds.
Key Results
- Effective Mitigation: The framework significantly reduced NSE penalties across both domains as the feedback budget increased.
- Feedback Efficiency: Using "Strict Approval" feedback (HA-S), agents achieved near-optimal safety with as few as 500 samples in complex environments.
- The Role of Slack: Experiments demonstrated that allowing a small amount of slack (allowing the agent to be ~15% less efficient) often led to a dramatic decrease in side effects, highlighting that strict efficiency often conflicts with safety.
Discussion and Insights
The authors argue that a lexicographic approach is superior to standard scalarization (combining rewards into a single weighted sum) because it avoids the difficulty of tuning weights between disparate units (e.g., time vs. dirtiness). Furthermore, the framework is robust enough to handle unavoidable NSEs by using the slack budget to minimize damage rather than failing the task.
Limitations and Future Work
Current limitations include the assumption that side effects correlate with existing state features and that the environment remains stationary. Future research will address "latent" or delayed side effects that are not immediately observable.
Summary updated