Towards Automating the Generation of Human-Robot Interaction Scenarios

Matthew C. Fontaine

Abstract

Evaluating Human-Robot Interaction (HRI) typically relies on hand-authored user studies, which are slow, expensive, and limited in scope. This paper proposes a framework to automate the generation of diverse and realistic HRI scenarios in simulation. By integrating Quality Diversity (QD) algorithms, generative models, and constraint solvers, the framework identifies critical failure modes that traditional testing might overlook. The approach ensures that generated environments are both stylistically realistic and functionally valid, providing a scalable alternative to manual scenario design.

Introduction

In the HRI community, the gold standard for evaluation is the user study. Researchers design specific tasks, observe human subjects interacting with robots, and draw conclusions. However, as robots move into complex, unconstrained environments like home kitchens, the "scenario space" grows exponentially. Manual testing can only cover a tiny fraction of possible interactions, leaving robots vulnerable to "black swan" failures—unexpected edge cases caused by specific environmental layouts or human behaviors.

The challenge lies in automatically discovering these diverse failure cases while ensuring the scenarios remain realistic. This work addresses the bottleneck of manual evaluation by creating a pipeline for automated, high-quality scenario generation.

Methods

The proposed framework utilizes a multi-staged approach to generate scenarios that are both diverse and functional:

  1. Quality Diversity (QD) Search: Instead of optimizing for a single worst-case scenario, the system uses MAP-Elites to find a diverse map of scenarios varying across specific measures, such as object distance or human "rationality."
  2. CMA-ME (Covariance Matrix Adaptation MAP-Elites): This algorithm improves search efficiency by using an adaptive sampling Gaussian, allowing it to navigate complex spaces faster than fixed-step methods.
  3. Generative Adversarial Networks (GANs): To maintain realism, the framework searches the latent space of a GAN trained on existing environment data to ensure generated levels match the style of human-designed spaces.
  4. Mixed Integer Programming (MIP) Repair: Because GANs can produce "broken" levels (e.g., unreachable goals), a MIP solver is used to "repair" environments, ensuring they satisfy hard functional constraints with minimal changes.
  5. Differentiable Quality Diversity (DQD): Recent iterations introduce CMA-MEGA, which leverages gradient information to navigate the latent spaces of deep models like StyleGAN more effectively.
The automated scenario generation pipeline combining CMA-ME, GANs, and MIP repair.

Key Insights and Results

The Environment as a First-Class Variable

A significant finding was that robot failures are often not the result of poor AI logic alone, but rather the physical layout of the world. In shared autonomy tasks, specific object placements can cause a robot to misinterpret human intent even when the human acts optimally. This highlights that the environment is just as critical to HRI success as the agent's underlying algorithms.

A shared autonomy task where the robot must infer the user's goal; specific layouts can lead to intent inference failures.

Performance and Validation

  • Efficiency: The CMA-ME algorithm more than doubled the performance of standard MAP-Elites across Quality Diversity metrics.
  • Solvability: The integration of MIP repair ensures that 100% of generated scenarios are functionally solvable, bridging the gap between aesthetic realism and practical utility.
  • Real-World Correlation: An online user study confirmed that environments inducing high workloads for simulated agents caused similar frustrations for real human teammates, validating simulation as a reliable proxy for real-world testing.

Conclusion and Future Work

While the current framework excels in grid-world environments like Overcooked, extending these methods to high-fidelity 3D environments remains a primary challenge. Future research aims to integrate Differentiable Quality Diversity (DQD) with models like OpenAI’s CLIP. This would allow researchers to guide scenario generation using natural language or complex visual concepts, making the automated search process more intuitive and powerful for HRI practitioners.

Summary updated

Abstract

My work studies the problem of generating scenarios to evaluate interaction between humans and robots. I expect these interactions to grow in complexity as robots become more intelligent and enter our daily lives. However, evaluating such interactions only through user studies, which are the de facto evaluation method in human-robot interaction, will quickly become infeasible as the number of possible scenarios grows exponentially with scenario complexity. Therefore, I propose automatically generating scenarios in simulation to explore the diverse possibility space of scenarios to better understand interaction and avoid costly failures in real world settings.
BibTeX
@inproceedings{fontaine2022towards,
  author = {Matthew C. Fontaine},
  title = {{Towards Automating the Generation of Human-Robot Interaction Scenarios}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21575},
}
Figure