Should Robots be Obedient?

Smitha Milli, Dylan Hadfield-Menell, Anca D. Dragan, Stuart Russell

Introduction: The Problem with Perfect Obedience

In AI safety and human-robot interaction (HRI), obedience is often treated as an unalloyed virtue. However, humans are "noisily rational"—we make mistakes, give vague commands, or change our minds under pressure. A robot that follows a literal but flawed order (e.g., a self-driving car turning into a ditch because a panicked passenger grabbed the wheel) is technically obedient but practically dangerous. This paper challenges the assumption that robots should always be perfectly obedient, proposing a shift toward "Do What I Mean" (DWIM) systems.

Framework: The Supervision POMDP

The authors formalize the interaction between an imperfect human and a learning robot using a Supervision POMDP (Partially Observable Markov Decision Process). In this model, the human knows the true reward parameters ($\theta$) but may act suboptimally. The robot does not know $\theta$ but observes the human's orders to infer it. This framework was tested using synthetic simulations where human behavior follows a Boltzmann (softmax) policy, and the authors provide a Jupyter notebook to replicate these experiments.

Comparison of blind obedience vs. intent inference.

At each step, the robot must decide whether to follow the order literally or take an alternative action based on its estimate of the user's true intent. The research compares several strategies for intent estimation:

  • Blind Obedience: Always follows the literal order.
  • Bayesian IRL (R*): Uses the posterior mean of $\theta$ to maximize expected value.
  • MLE-R: Uses a Maximum Likelihood Estimate, which proves to be more "cautious" and obedient during early learning stages, effectively giving the human the "benefit of the doubt."

Key Insights: Rationality and Model Misspecification

A core finding is the Rationality Gap: if humans were perfectly rational, obedience would be optimal. Disobedience is only beneficial because humans are imperfect.

However, autonomy comes with risks. The authors highlight the danger of Missing Features. If a robot's model lacks critical features (e.g., it doesn't recognize a "wet floor" as a hazard), it might confidently disobey a correct order, leading to "rebellion regret." Interestingly, having extra irrelevant features makes a robot safer and more obedient than having too few. To mitigate risks, the authors propose a "burn-in" period: if a robot cannot find any reward function that explains the user's initial orders, it concludes its model is misspecified and reverts to blind obedience.

Key Results: The Autonomy-Obedience Trade-off

The paper proves that an autonomous robot can always outperform a blindly obedient one as long as the human is not perfectly rational. Quantitative simulations show a clear trade-off: as the robot learns more about the user's preferences, its autonomy advantage ($\Delta$)—the extra value provided by acting on inferred intent—increases, while its literal obedience ($O$) decreases.

Autonomy advantage and obedience trends over interaction steps.

Furthermore, while the "optimal" Bayesian robot is sensitive to the exact level of human noise, the MLE-R approach is robust, maintaining consistent performance regardless of whether it overestimates or underestimates human irrationality.

Conclusion and Future Directions

Moving beyond literal obedience allows robots to better serve imperfect human users. However, this requires sophisticated intent inference and robust handling of model misspecification to avoid dangerous "rebellion." Future work should explore more complex human behaviors beyond simple noisy rationality and investigate game-theoretic scenarios where humans might intentionally provide instructional orders to a learning robot.

Summary updated

Abstract

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.
BibTeX
@inproceedings{milli2017should,
  author = {Smitha Milli and Dylan Hadfield-Menell and Anca D. Dragan and Stuart Russell},
  title = {{Should Robots be Obedient?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/662},
}
Figure