ICRA Roboethics Challenge 2023: Intelligent Disobedience in an Elderly Care Home

Sveta Paster, Kantwon Rogers, Gordon Briggs, Peter Stone, Reuth MirskyarXiv 2023 · DOI · arXiv

Introduction

As service robots become increasingly integrated into elderly care facilities, the traditional paradigm of "blind obedience" to user commands presents significant ethical risks. While Isaac Asimov’s Second Law suggests robots must obey human orders, real-world caregiving is rife with ethical minefields where simple obedience can lead to harm. In an elderly care setting, a robot might be asked to fetch items that compromise privacy (another resident's medical records), safety (a knife for a resident with a mental condition), or health (prohibited substances).

A service robot interacting with a resident in a care home setting.

The challenge lies in creating a robot that can respect a resident’s autonomy while adhering to global safety objectives, all while maintaining transparency so the user understands why a request was denied. This paper proposes a five-step "Intelligent Disobedience" framework that allows robots to deliberate on the implications of a request and provide personalized, ethical alternatives instead of simple refusal.

The Intelligent Disobedience Framework

The proposed framework adapts a paradigm originally derived from the training of guide dogs, who are taught to ignore a "forward" command if it leads into traffic. The robotic implementation follows five distinct stages:

  1. Global Objectives: Defining constant, high-level goals, such as "keep the resident healthy and safe." The authors suggest using Inverse Reinforcement Learning (IRL) to help robots balance these often-conflicting constraints.
  2. Local Objectives: Eliciting the user's immediate desire through Natural Language Processing (NLP).
  3. Plan Recognition: Using "Theory of Mind" to infer hidden intentions. For instance, if a resident asks for a specific bag, the robot must infer the underlying plan (e.g., smoking) rather than just the literal action (fetching a bag).
  4. Consistency Check: The robot evaluates if the recognized plan conflicts with the global objective (e.g., respiratory health).
  5. Mediation: If a conflict is found, the robot attempts to resolve the issue by choosing a personalized response—such as suggesting chewing gum or meditation—based on the resident's specific medical and psychological profile.
The five-step logic used by the robot to evaluate and mediate conflicting user requests.

Key Insights and Ethical Considerations

The Transparency Paradox

While transparency is usually a goal in AI, it can conflict with privacy in care settings. Explaining exactly why a robot won't fetch an item might inadvertently reveal confidential information about another resident. Furthermore, the robot’s response must be context-dependent; refusing a cigarette to a resident with lung disease requires therapeutic mediation, whereas refusing the same to a healthy visitor might only require a statement of facility rules.

Theory of Mind

Ethical robotics is not merely about parsing verbs; it is about understanding the "why" behind a request. Without sophisticated plan recognition, a robot cannot detect the harm hidden behind a seemingly benign request. This requires the robot to have the agency to say "no"—an "ethical rebellion" necessary to protect its user.

Evaluation and Future Work

As a framework proposal for the ICRA Roboethics Challenge, the authors focus on qualitative formalization. They successfully mapped a "Bag Fetching" scenario onto the five-step framework and proposed evaluation metrics centered on time-to-resolution and social satisfaction (measured via the Godspeed questionnaire), rather than simple task completion.

Future work must address potential value misalignments, where a resident prioritizes autonomy over long-term health. Additionally, the framework needs to account for designer bias and the complexities of multi-user environments where different residents may have conflicting objectives. If the robot’s internal model is incorrect, "intelligent" disobedience risks becoming an "ignorant" hindrance, necessitating further research into handling uncertainty during plan recognition.

Summary updated

Abstract

With the projected surge in the elderly population, service robots offer a promising avenue to enhance their well-being in elderly care homes. Such robots will encounter complex scenarios which will require them to perform decisions with ethical consequences. In this report, we propose to leverage the Intelligent Disobedience framework in order to give the robot the ability to perform a deliberation process over decisions with potential ethical implications. We list the issues that this framework can assist with, define it formally in the context of the specific elderly care home scenario, and delineate the requirements for implementing an intelligently disobeying robot. We conclude this report with some critical analysis and suggestions for future work.
BibTeX
@misc{paster2023icra,
  author = {Sveta Paster and Kantwon Rogers and Gordon Briggs and Peter Stone and Reuth Mirsky},
  title = {{ICRA Roboethics Challenge 2023: Intelligent Disobedience in an Elderly Care Home}},
  year = {2023},
  eprint = {2311.08783},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.08783},
}
Figure