Get the Ball Rolling: Alerting Autonomous Robots When to Help to Close the Healthcare Loop

Jiaxin Shen, Yanyao Liu, Ziming Wang, Ziyuan Jiao, Yufeng Chen, Wenjuan HanarXiv 2023 · DOI · OpenReview

Introduction: The Shift to Proactive Healthcare Robotics

Current robotic systems in healthcare are primarily reactive, relying on explicit voice commands or pre-defined triggers to initiate action. However, in critical scenarios—such as an elderly person fainting or falling—the individual may be unable to call for help. This research introduces the Autonomous Helping Challenge, which pivots the focus from following instructions to proactively identifying when assistance is required. By shifting the burden of task initiation from the human to the robot, the "healthcare loop" can be effectively closed even in emergencies.

Example of a person in distress (fallen) that the robot must autonomously identify as needing help.

The authors identify three primary hurdles in current systems: the Autonomous Task Generation gap (deciding when to act), the Commonsense Gap (interpreting visual distress as a need for intervention based on static knowledge), and the Grounding Gap (translating abstract goals into precise physical robot motions in messy environments).

The Helpy Framework

To address these gaps, the researchers propose Helpy, a "learning-free" framework that leverages the reasoning capabilities of pre-trained foundation models like Vision-Language Models (VLMs) and Large Language Models (LLMs). Unlike traditional methods that require extensive training on specific tasks, Helpy uses these models as zero-shot reasoners and code generators.

The Helpy framework closes the healthcare loop through a cycle of task generation, planning, and feedback.

The Helpy system operates in a continuous loop consisting of four main stages:

  1. Altruistic Task Generation: A VLM (e.g., MiniGPT-4) monitors the environment, prompted to act as a "helpful person." It is asked, "Do you think you should offer extra help?" If it detects a crisis, it uses Grounded DINO to localize and box the person in distress.
  2. Task Planning: An LLM (GPT-3.5) decomposes the high-level goal (e.g., "help the elder") into structured Pythonic functions (e.g., hold(arm, arm_pose, agent)). This allows for a more structured and executable plan than natural language alone.
  3. Motion Planning: These functions are translated into 6-DoF trajectories for a physical mobile manipulator, specifically a Clearpath Ridgeback base equipped with UR5 arms.
  4. Environmental Feedback: The system monitors the environment in real-time, updating its internal state. This allows it to generate new tasks or adjust plans if the initial intervention fails or the situation evolves.

The Autonomous Helping Dataset

A significant contribution of this work is the Autonomous Helping dataset, a large-scale multi-modal collection designed to train and evaluate robot "alertness." The dataset is comprised of:

  • Video Data: 55 videos containing 37,157 RGB-D frames across 13 motion categories.
  • Multi-modal Inputs: Data includes First-Person View (FPV), Third-Person View (TPV), audio, IMU data, depth maps, and human poses.
  • Image Data: Over 12,000 images sourced from public datasets and web crawling, covering diverse emergency scenarios like falls, fires, car crashes, and natural disasters.

Results and Key Insights

The framework was evaluated on a physical robot across varying task complexities. While the overall Success Rate (SR) of 25.59% highlights the extreme difficulty of autonomous physical intervention, Helpy demonstrated significant reasoning advantages. Success rates varied by difficulty, dropping from 30.1% for "Easy" tasks to 12.3% for "Hard" tasks.

In human-judged effectiveness, Helpy scored 73/100, significantly outperforming the previous state-of-the-art, ProgPrompt (29/100). A key insight is the use of LLMs as code generators, allowing the robot to handle open-ended scenarios without task-specific training. However, the "No Harm" baseline—the ability to correctly identify when not to intervene—remains a challenge, with foundation models scoring 64.7%, indicating a need for better nuance in interpreting human social cues.

Limitations and Future Directions

The research highlights several critical limitations, including error propagation (where early VLM misinterpretations derail the entire plan) and physical reliability (e.g., gripper slippage during a lift). Future work aims to unify LLMs and VLMs into a single multi-modal foundation model to bridge the "translation" gap between visual perception and physical execution.

Summary updated

Abstract

To facilitate the advancement of research in healthcare robots without human intervention or commands, we introduce the Autonomous Helping Challenge, along with a crowd-sourcing large-scale dataset. The goal is to create healthcare robots that possess the ability to determine when assistance is necessary, generate useful sub-tasks to aid in planning, carry out these plans through a physical robot, and receive feedback from the environment in order to generate new tasks and continue the process. Besides the general challenge in open-ended scenarios, Autonomous Helping focuses on three specific challenges: autonomous task generation, the gap between the current scene and static commonsense, and the gap between language instruction and the real world. Additionally, we propose Helpy, a potential approach to close the healthcare loop in the learning-free setting.
BibTeX
@misc{shen2023get,
  author = {Jiaxin Shen and Yanyao Liu and Ziming Wang and Ziyuan Jiao and Yufeng Chen and Wenjuan Han},
  title = {{Get the Ball Rolling: Alerting Autonomous Robots When to Help to Close the Healthcare Loop}},
  year = {2023},
  eprint = {2311.02602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.02602},
}
Figure