Evaluating the Pertinence of Robot Decisions in a Human-Robot Joint Action Context: The PeRDITA Questionnaire

Sandra Devin, Camille Vrignaud, Kathleen Belhassein, Aurélie Clodic, Ophélie Carreras, Rachid Alami2018 · DOI

TL;DR

This paper introduces PeRDITA, a novel questionnaire designed to evaluate how well robots make decisions during physical human-robot joint actions. By focusing on the pertinence of these decisions, PeRDITA aims to provide a standardized way to measure progress in this crucial area of HRI, moving beyond general interaction or basic task evaluations.

What Problem Does This Paper Address?

As robots become more integrated into collaborative tasks with humans, the ability of robots to make intelligent, context-aware decisions is paramount. While existing questionnaires assess general human-robot interaction (HRI) or specific robot attributes, there's a significant gap in tools that specifically evaluate the pertinence of a robot's high-level decisions within a physical joint action context. This includes decisions about what action to perform, who should perform it, and when, especially in complex scenarios where shared goals and resources are involved. Without such a tool, it's difficult to quantitatively measure and compare advancements in robot decision-making for collaborative tasks.

Methods

The authors developed the PeRDITA questionnaire based on User Experience (UX) models, specifically focusing on instrumental qualities (usability, functionality) and non-instrumental qualities (hedonic attributes, pleasure derived from use). The questionnaire is structured around several dimensions:

  • Interaction: Evaluates the overall behavioral intention to use the system.
  • Robot Perception: Assesses how the human perceives the robot's general attributes.
  • Verbal: Measures the perceived quality of the robot's verbal communication.
  • Acting: Assesses how the human perceives the robot's decision-making regarding its actions.
  • Collaboration: Evaluates the perceived acceptability, usability, and security of the collaboration.

PeRDITA uses antonym items with a 100-point semantic differential scale for subjects to rate their experience. The paper also describes a pilot study where subjects performed a joint task with a PR2 robot under different system conditions to test the questionnaire's validity and reliability.

Datasets

Not discussed in the paper. The study utilized a custom-built experimental setup and task for evaluation.

Key Insights and Ideas

  • The Need for Specificity: General HRI questionnaires are insufficient for evaluating the nuanced decision-making required in physical joint actions. PeRDITA's focus on "pertinence" addresses this gap.
  • Decision-Making Complexity: Evaluating robot decisions is challenging because it's hard to isolate the decisional aspect from other robot abilities (perception, manipulation) and because naive users may not easily distinguish subtle differences in robot behavior.
  • The "Annoyance" Factor: While users might initially find verbose robots comforting, this can quickly become annoying, highlighting the difficulty of evaluating verbal interventions in short-term studies.
  • New System Preference: A new system that incorporates estimations of the human's mental state and defers some decisions to the last moment was significantly preferred by users over a system that made all decisions in advance. This suggests that more flexible and human-aware decision-making is highly valued.

Key Results

The pilot study revealed that the new system (NS), which incorporated human mental state estimation and deferred decision-making, was significantly preferred over the reference system (RS) across most dimensions, including the overall score (p < 0.05). The "Acting" dimension showed a particularly strong preference for the new system (p ~ 0.003).

However, the comparison between negotiation and adaptation modes within the new system did not yield significant differences, except for the "Verbal" dimension, which had questionable internal consistency. The "Verbal" dimension itself showed low reliability (Cronbach's alpha = 0.6), suggesting it needs further refinement.

Limitations and Future Work

The authors acknowledge several limitations:

  • Task Specificity: The chosen task, while designed to elicit decision-making, might not fully capture all aspects of human-robot joint action.
  • Naive Users: The study relied on naive users, whose perceptions might differ from those of experienced users or in long-term interactions. Their initial positive reactions might be due to novelty rather than true long-term utility.
  • Short-Term Evaluation: The difficulty in evaluating aspects like "annoyance" or "repetitiveness" in short-term studies is a significant challenge.
  • Verbal Dimension Reliability: The "Verbal" dimension of the questionnaire needs reconsideration and validation on different tasks.

Future work includes refining the PeRDITA questionnaire, particularly the "Verbal" dimension, and testing it on other tasks and contexts. The authors also suggest exploring methods for long-term user studies that are feasible within research constraints, potentially involving more experienced users. They also plan to replace the "Robot Perception" dimension with a "Perceived Competence of the Robot" dimension, believing it to be more directly relevant to evaluating decision pertinence.

Summary updated

Abstract

International audience
BibTeX
@misc{devin2018evaluating,
  author = {Sandra Devin and Camille Vrignaud and Kathleen Belhassein and Aurélie Clodic and Ophélie Carreras and Rachid Alami},
  title = {{Evaluating the Pertinence of Robot Decisions in a Human-Robot Joint Action Context: The PeRDITA Questionnaire}},
  year = {2018},
  doi = {10.1109/roman.2018.8525785},
}

Loading related content...

Figure