Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summaries

Peter Du, Surya Murthy, Katherine Rose Driggs-CampbellarXiv 2023 · DOI · arXiv

Abstract

As autonomous agents become increasingly sophisticated, evaluating their performance through random observation becomes inefficient, especially when comparing high-performing models. This paper introduces an Adaptive Scenario Search framework that automatically identifies "stress test" scenarios where two robot policies diverge significantly in behavior. By presenting these contrasting summaries to human observers, the method enables accurate identification of superior agents with up to 95% accuracy, even when performance differences are subtle.

Introduction

Trusting and understanding complex autonomous systems, such as self-driving cars or delivery robots, requires more than observing routine operations. Standard behavior summarization techniques often rely on random sampling, which fails to capture critical "edge cases" where a robot's true capabilities are tested. In scenarios where two robots perform similarly in 99% of situations, random footage is uninformative. This research addresses the challenge of efficiently finding informative scenarios for robots with continuous action spaces to help humans quickly discern performance hierarchies.

Methodology

The proposed framework, Adaptive Scenario Search, treats the discovery of informative summaries as a reinforcement learning problem. The system architecture consists of several key components:

  1. Coupled Simulation: Two distinct robot policies are executed within the same simulated environment, ensuring they encounter identical external stimuli.
  2. Divergence-based Reward: The search is guided by a reward function that prioritizes scenarios resulting in high action divergence (different steering/acceleration) and state divergence (different final positions).
  3. MCTS Solver: A Monte Carlo Tree Search (MCTS) algorithm explores the environment's action space to find sequences that maximize the divergence between the two agents.
Overview of the Adaptive Scenario Search framework, featuring the Coupled Simulator and MCTS Solver.

Experimental Setup

The framework was evaluated using a Robot Crowd Navigation simulator. The agents utilized a "Decentralized Structural-RNN" (DS-RNN) architecture trained via Deep Reinforcement Learning. Three versions of the agent—Low ($A^{(lo)}$), Medium ($A^{(med)}$), and High ($A^{(hi)}$) performance—were created based on training duration, ranging from $5.0 \times 10^6$ to $1.2 \times 10^7$ timesteps. A user study involving 40 participants compared the effectiveness of adaptive summaries against a baseline "N-First" approach, which simply shows the first few random scenarios encountered.

Results and Discussion

The study demonstrated that "agreement is uninformative." Information-dense moments occur when robot paths diverge. As robots improve, they appear identical in most situations, making adversarial-style search essential for evaluation.

Quantitative results highlight the superiority of the adaptive approach across both pairwise and global ranking tasks:

  • Pairwise Accuracy: Participants correctly identified the better robot 95% of the time using adaptive summaries, compared to only 54.17% for the baseline.
  • Global Ranking: Accuracy for ranking three robots improved from 30% (baseline) to 65% (adaptive).
  • User Experience: Participants reported that contrasting summaries made the task "Very Easy," whereas baseline scenarios often left them unable to distinguish between agents due to identical-looking behaviors.
Table comparing the accuracy of Pairwise and Global rankings between N-First Trajectories and Adaptive Search.

Conclusion

While computationally intensive due to the need for pairwise searches, Adaptive Scenario Search significantly enhances human ability to evaluate autonomous agents. Future work aims to scale the framework to multi-agent comparisons and apply it to diverse domains like flight control and industrial robotics.

Summary updated

Abstract

As advances in artificial intelligence enable increasingly capable learning-based autonomous agents, it becomes more challenging for human observers to efficiently construct a mental model of the agent's behaviour. In order to successfully deploy autonomous agents, humans should not only be able to understand the individual limitations of the agents but also have insight on how they compare against one another. To do so, we need effective methods for generating human interpretable agent behaviour summaries. Single agent behaviour summarization has been tackled in the past through methods that generate explanations for why an agent chose to pick a particular action at a single timestep. However, for complex tasks, a per-action explanation may not be able to convey an agents global strategy. As a result, researchers have looked towards multi-timestep summaries which can better help humans assess an agents overall capability. More recently, multi-step summaries have also been used for generating contrasting examples to evaluate multiple agents. However, past approaches have largely relied on unstructured search methods to generate summaries and require agents to have a discrete action space. In this paper we present an adaptive search method for efficiently generating contrasting behaviour summaries with support for continuous state and action spaces. We perform a user study to evaluate the effectiveness of the summaries for helping humans discern the superior autonomous agent for a given task. Our results indicate that adaptive search can efficiently identify informative contrasting scenarios that enable humans to accurately select the better performing agent with a limited observation time budget.
BibTeX
@misc{du2023conveying,
  author = {Peter Du and Surya Murthy and Katherine Rose Driggs-Campbell},
  title = {{Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summaries}},
  year = {2023},
  eprint = {2304.00367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.00367},
}
Figure