Introduction: The Problem with Perfect Obedience
In AI safety and human-robot interaction (HRI), obedience is often treated as an unalloyed virtue. However, humans are "noisily rational"—we make mistakes, give vague commands, or change our minds under pressure. A robot that follows a literal but flawed order (e.g., a self-driving car turning into a ditch because a panicked passenger grabbed the wheel) is technically obedient but practically dangerous. This paper challenges the assumption that robots should always be perfectly obedient, proposing a shift toward "Do What I Mean" (DWIM) systems.
Framework: The Supervision POMDP
The authors formalize the interaction between an imperfect human and a learning robot using a Supervision POMDP (Partially Observable Markov Decision Process). In this model, the human knows the true reward parameters ($\theta$) but may act suboptimally. The robot does not know $\theta$ but observes the human's orders to infer it. This framework was tested using synthetic simulations where human behavior follows a Boltzmann (softmax) policy, and the authors provide a Jupyter notebook to replicate these experiments.
At each step, the robot must decide whether to follow the order literally or take an alternative action based on its estimate of the user's true intent. The research compares several strategies for intent estimation:
- Blind Obedience: Always follows the literal order.
- Bayesian IRL (R*): Uses the posterior mean of $\theta$ to maximize expected value.
- MLE-R: Uses a Maximum Likelihood Estimate, which proves to be more "cautious" and obedient during early learning stages, effectively giving the human the "benefit of the doubt."
Key Insights: Rationality and Model Misspecification
A core finding is the Rationality Gap: if humans were perfectly rational, obedience would be optimal. Disobedience is only beneficial because humans are imperfect.
However, autonomy comes with risks. The authors highlight the danger of Missing Features. If a robot's model lacks critical features (e.g., it doesn't recognize a "wet floor" as a hazard), it might confidently disobey a correct order, leading to "rebellion regret." Interestingly, having extra irrelevant features makes a robot safer and more obedient than having too few. To mitigate risks, the authors propose a "burn-in" period: if a robot cannot find any reward function that explains the user's initial orders, it concludes its model is misspecified and reverts to blind obedience.
Key Results: The Autonomy-Obedience Trade-off
The paper proves that an autonomous robot can always outperform a blindly obedient one as long as the human is not perfectly rational. Quantitative simulations show a clear trade-off: as the robot learns more about the user's preferences, its autonomy advantage ($\Delta$)—the extra value provided by acting on inferred intent—increases, while its literal obedience ($O$) decreases.
Furthermore, while the "optimal" Bayesian robot is sensitive to the exact level of human noise, the MLE-R approach is robust, maintaining consistent performance regardless of whether it overestimates or underestimates human irrationality.
Conclusion and Future Directions
Moving beyond literal obedience allows robots to better serve imperfect human users. However, this requires sophisticated intent inference and robust handling of model misspecification to avoid dangerous "rebellion." Future work should explore more complex human behaviors beyond simple noisy rationality and investigate game-theoretic scenarios where humans might intentionally provide instructional orders to a learning robot.
Summary updated