Should Robots be Obedient?

Smitha Milli, Dylan Hadfield-Menell, Anca D. Dragan, Stuart Russell

TL;DR

This paper argues that blindly obedient robots aren't always optimal, especially when humans are irrational. They propose that robots should learn human preferences to act more intelligently, leading to a trade-off between obedience and achieving the human's true goals. The authors also explore how model misspecifications impact this trade-off and suggest methods for robots to detect and adapt to them.

What Problem Does This Paper Address?

The paper tackles the fundamental question of whether robots should always obey human commands. While intuitive obedience seems desirable for safety and usability, the authors highlight a critical flaw: humans themselves are not perfectly rational and can issue suboptimal commands. This creates a scenario where a robot blindly following an order might actually lead to a worse outcome than if it had understood the human's underlying intent. The core problem is to formalize this trade-off between obedience and achieving the human's true, potentially unstated, preferences, and to understand how different robot learning strategies and model assumptions affect this balance.

Methods

The authors introduce a formal model of human-robot interaction, framing it as a supervision POMDP (Partially Observable Markov Decision Process). They then simplify this to a repeated game for analysis. The key technical contribution lies in exploring Inverse Reinforcement Learning (IRL) as a mechanism for the robot (R) to infer the human's (H) reward parameters ($\theta$).

Instead of simply following orders, an IRL-R attempts to estimate $\theta$ from H's commands and then acts to maximize the expected reward based on this estimate. This contrasts with a blindly obedient robot that always follows H's literal order. The paper analyzes different IRL approaches, including Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP), and their impact on the robot's obedience and the autonomy advantage ($\Delta$), which is the difference between the reward achieved by the robot acting on inferred preferences and the reward from blindly following orders. They also investigate the robustness of these methods to misspecified models of human rationality ($\beta$) and preferences (feature representations).

Datasets

Not discussed in the paper. The experiments are simulated and rely on synthetic data generated according to the defined models.

Key Insights and Ideas

  • Blind obedience is not always best: When humans are irrational, a robot that infers preferences can achieve higher rewards than one that blindly follows orders. This is because the robot can correct for the human's suboptimal decision-making.
  • The Obedience-Autonomy Trade-off: There's an inherent trade-off: higher autonomy (achieving more of the human's true goals) often comes at the cost of lower obedience (deviating from literal commands).
  • IRL as a Solution: Inverse Reinforcement Learning provides a principled way for robots to learn human preferences and act more intelligently, balancing obedience with goal achievement.
  • MLE-R is Robust and Obedient: The Maximum Likelihood Estimate (MLE) for inferring human preferences is shown to be robust to misspecifications in the human's rationality parameter ($\beta$) and tends to err on the side of obedience, especially for the first order. This makes it a good practical approximation.
  • Model Misspecification Matters: Incorrect models of human preferences (e.g., missing features) can lead to a robot being less obedient than it should be, potentially resulting in negative autonomy advantage (rebellion regret). Conversely, having extra features can make the robot more obedient and still achieve good autonomy.
  • Detecting Misspecification: Robots can potentially detect missing features by observing their own obedience rates, particularly by using a "burn-in" period where they initially obey more to gather data.

Key Results

  • Positive Autonomy Advantage: When humans are not perfectly rational, an IRL robot can always achieve a positive autonomy advantage ($\Delta > 0$) compared to a blindly obedient robot.
  • Trade-off Illustrated: Figures 2a and 2b empirically demonstrate the trade-off between autonomy advantage ($\Delta$) and obedience ($O$). As the robot learns and becomes more autonomous, its obedience decreases.
  • Impact of Human Irrationality: Figure 3 shows that as human irrationality ($\beta$) increases, the autonomy advantage converges to a higher value, though at a slower rate.
  • MLE-R Robustness: Theorem 5 shows that MLE-R is robust to misspecified human rationality ($\beta$), meaning its actions remain the same even if its model of $\beta$ is wrong. The optimal policy, however, is not.
  • Feature Misspecification: Figure 4 illustrates that missing features in the robot's model lead to lower autonomy advantage and potentially negative values, while added features increase obedience and maintain positive autonomy.

Limitations and Future Work

  • Model Assumptions: The paper relies on a simplified repeated game model and assumes a linear reward function. Real-world scenarios might involve more complex state transitions and reward structures.
  • Computational Cost: While MLE is presented as a more robust approximation, Bayesian IRL (which the optimal policy approximates) can be computationally expensive in complex environments.
  • Human Rationality Models: The paper primarily considers rational and noisily rational humans. Future work could explore more complex models of human behavior, such as inconsistency or false beliefs.
  • Detecting Misspecification: While the paper proposes a method for detecting missing features, further research is needed to develop robust methods for detecting other types of model misspecification.
  • Long-term Goal: The authors advocate for robots that can intelligently decide when to obey, but acknowledge that in the interim, ensuring robustness and erring on the side of obedience is crucial. Future work should focus on developing these intelligent decision-making capabilities.

Summary updated

Abstract

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.
BibTeX
@inproceedings{milli2017should,
  author = {Smitha Milli and Dylan Hadfield-Menell and Anca D. Dragan and Stuart Russell},
  title = {{Should Robots be Obedient?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/662},
}
Figure