Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery

Mimo Shirasaka, Tatsuya Matsushima, Soshi Tsunashima, Yuya Ikeda, Aoi Horo, So Ikoma, Chikaha Tsuji, Hikaru Wada, Tsunekazu Omija, Dai Komukai, Yutaka Matsuo, Yusuke Iwasawa

TL;DR

This paper introduces a promptable General Purpose Service Robot (GPSR) system that leverages multiple foundation models for enhanced generalization and adaptability. To address real-world failures, they propose a "self-recovery prompting pipeline" that allows the robot to autonomously recover from insufficient information, incorrect plan generation, or execution failures by dynamically adjusting its prompts.

What Problem Does This Paper Address?

Developing a truly general-purpose service robot (GPSR) capable of handling diverse tasks in unpredictable real-world environments remains a significant challenge. While foundation models have shown promise in improving robot perception and planning, current systems often struggle with the inherent ambiguity and dynamism of human environments. Specifically, robots can fail due to: 1) insufficient information (e.g., "find my watch" when the robot has no idea where to look), 2) incorrect plan generation (e.g., misinterpreting a command and creating a nonsensical sequence of actions), or 3) plan execution failure (e.g., a physical inability to grasp an object). Existing approaches often treat these as separate issues or rely on human intervention, hindering true autonomy. This paper aims to create a more robust GPSR system that can autonomously recover from these common failure modes.

Methods

The core of the system is a promptable GPSR architecture built upon several foundation models:

  • Whisper for speech recognition.
  • GPT-4 for task planning.
  • Detic and CLIP for object detection and classification.
  • CLIP-Fields for semantic mapping.

The novelty lies in the self-recovery prompting pipeline. This pipeline enhances the foundation-model-based system by incorporating mechanisms to address the three identified failure modes:

  1. Insufficient Information (M1): The system uses LLMs to infer likely locations based on common sense (e.g., food in the kitchen) or actively queries the human operator for clarification.
  2. Incorrect Plan Generation (M2): The system improves command recognition by tuning Whisper's prompts with task-specific knowledge. For plan generation, it refines GPT-4's prompts based on human feedback or examples of successful task completions.
  3. Plan Execution Failure (M3): For per-skill recovery, the system can retry a failed skill or replan an alternative skill sequence. For per-plan recovery (when the entire task is deemed a failure), it updates prompts for object recognition and regenerates the entire plan.

Datasets

The paper does not explicitly mention training datasets for the foundation models themselves, as they are pre-trained. However, the system was evaluated in a real-world simulated household environment. For experiments, they used a set of seven handcrafted commands designed to trigger specific failure modes.

Key Insights and Ideas

  • Prompting is Key to Adaptability: The paper strongly emphasizes that by carefully crafting prompts, foundation models can be adapted to specific tasks and environments without retraining, making the system highly flexible.
  • Failure Modes are Categorizable and Addressable: Identifying and categorizing failures into insufficient information, incorrect plan generation, and execution failure provides a structured approach to developing recovery mechanisms.
  • Self-Recovery via Prompt Engineering: The core innovation is demonstrating that self-recovery can be achieved by dynamically adjusting prompts to foundation models, rather than requiring complex new model architectures or extensive retraining. This makes the recovery process more efficient and integrated.
  • Real-World Performance Matters: The authors highlight that success in competitions like RoboCup, while valuable, doesn't fully capture the requirements of a true GPSR. Real-world robustness against failures is crucial.

Key Results

The proposed promptable GPSR system achieved high performance in competitions, winning first prize in the GPSR task at RoboCup Japan Open 2023 and second place overall. Crucially, the self-recovery mechanism was experimentally confirmed to successfully recover from all seven handcrafted commands designed to induce the three failure modes. For instance, the system could recover from missing location information by asking for clarification or inferring it, and it could re-plan when faced with transcription difficulties or execution errors.

Limitations and Future Work

The authors acknowledge that the GPSR task in competitions abstracts away some real-world complexities, such as providing object information upfront. While their system demonstrates robustness in simulated environments, further validation in more diverse and dynamic real-world scenarios is needed. Future work should focus on developing benchmarks that incorporate adaptive human-robot interaction to standardize performance evaluation for self-recovering GPSR systems, potentially leveraging LLMs and VLMs further.

Summary updated

Abstract

A general-purpose service robot (GPSR), which can execute diverse tasks in various environments, requires a system with high generalizability and adaptability to tasks and environments. In this paper, we first developed a top-level GPSR system for worldwide competition (RoboCup@Home 2023) based on multiple foundation models. This system is both generalizable to variations and adaptive by prompting each model. Then, by analyzing the performance of the developed system, we found three types of failure in more realistic GPSR application settings: insufficient information, incorrect plan generation, and plan execution failure. We then propose the self-recovery prompting pipeline, which explores the necessary information and modifies its prompts to recover from failure. We experimentally confirm that the system with the self-recovery mechanism can accomplish tasks by resolving various failure cases. Supplementary videos are available at https://sites.google.com/view/srgpsr .
BibTeX
@inproceedings{shirasaka2023self,
  author = {Mimo Shirasaka and Tatsuya Matsushima and Soshi Tsunashima and Yuya Ikeda and Aoi Horo and So Ikoma and Chikaha Tsuji and Hikaru Wada and Tsunekazu Omija and Dai Komukai and Yutaka Matsuo and Yusuke Iwasawa},
  title = {{Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra57147.2024.10611640},
}
Figure