VADER: Visual Affordance Detection and Error Recovery for Multi Robot Human Collaboration

Michael Ahn, Montserrat Gonzalez Arenas, Matthew Bennice, Noah Brown, Christine Chan, Byron David, Anthony Francis, Gavin Gonzalez, Rainer Hessmer, Tomas Jackson, Nikhil J Joshi, Daniel Lam, Tsang-Wei Edward Lee, Alex Luong, Sharath Maddineni, Harsh Patel, Jodilyn Peralta, Jornell Quiambao, Diego Reyes, Rosario M Jauregui Ruano, Dorsa Sadigh, Pannag Sanketi, Leila Takayama, Pavel Vodenski, Fei Xia

TL;DR

VADER (Visual Affordance Detection and Error Recovery) is a framework that enables robots to complete complex, long-horizon tasks by adding a "Detect" step to the standard "Plan-Execute" loop. By using Visual Question Answering (VQA) to identify physical failures or missing capabilities, robots can autonomously decide to seek help from humans or other robots via a cloud-based orchestration service.

What Problem Does This Paper Address?

Large Language Model Planners (LMPs) have revolutionized how robots chain simple skills (like "pick up can") into complex tasks (like "clean the kitchen"). However, these plans often lack "grounding"—the robot might plan to wipe a table without realizing its gripper is broken, or it might fail a step and keep trying the next one as if nothing went wrong.

Current systems typically assume a single robot is working in a static environment. If that robot encounters a situation outside its physical capabilities—such as a wiping robot needing to move a heavy obstacle it can't grasp—the task simply fails. VADER addresses this by treating "seeking help" as a first-class skill, allowing robots to recognize their own limitations and collaborate to recover from errors.

Methods

VADER introduces a Plan-Execute-Detect loop that interleaves high-level reasoning with real-world verification:

  1. Plan: A Language Model Planner (LMP), powered by PaLM, selects the next skill and generates a natural language description of the expected outcome (e.g., "coke can in hand").
  2. Execute: The robot performs the skill using its low-level policies (trained via Reinforcement Learning or Behavioral Cloning).
  3. Detect: A VQA module (like PaLI or CLIP) inspects the new state. It compares the visual evidence against the expected outcome to detect errors or checks "affordances" (pre-conditions) to see if the next step is even possible.
  4. Recover: If a failure or missing affordance is detected, the feedback is fed back into the LMP. The planner then decides whether to retry, skip, or—crucially—request help.

To facilitate this help-seeking, the authors developed the Human Robot Fleet Orchestration Service (HRFS). This is a cloud-based system where robots can post sub-tasks (e.g., "please clear the table") that can be claimed by other robots with different tools or by nearby humans.

Fig. 1: VADER: Visual Affordance Detection and Error Recovery. A plan, execution, detect framework with a seeking help skill as a recovery mechanism. While executing a plan, the robot detects a deviation from its expectations – a coke can obstructing the table area to be wiped. It replans for recovery, and upon receiving help, completes the original plan.

Datasets

The researchers did not introduce a new training dataset but rather evaluated VADER in real-world office kitchen environments. They utilized several pre-trained state-of-the-art models:

  • LMP: PaLM for high-level planning.
  • VQA: Evaluated CLIP, ViLD, and PaLI for visual grounding.
  • Policies: A library of pre-computed policies for navigation, picking, and wiping.

The evaluation consisted of a robot-robot collaboration pilot study and a human-robot interaction (HRI) user study with 19 participants.

Key Insights and Ideas

  • Help as a Primitive: By framing "ask for help" as just another skill in the library, the LLM can naturally integrate collaboration into its recovery plans without hard-coded logic for every possible failure.
  • VQA for Grounding: Using VQA to verify outcomes provides a more flexible and policy-agnostic way to detect errors than traditional methods like checking RL value functions, which may not be available for all skills.
  • Morphological Awareness: In the pilot study, robots were not told what tools they had. They had to perform a "Self-Inspection" skill, using VQA to look at their own end-effectors to determine if they were a "wiper" or a "gripper," demonstrating a high level of embodied reasoning.
  • Social Perception: The user study revealed that robots asking for help aren't necessarily seen as "less capable." Instead, users felt more successful and collaborative when the robot communicated its needs.

Key Results

  • Collaborative Success: In the pilot study, two robots with different tools (one for wiping, one for gripping) successfully completed a table-cleaning task that neither could do alone in 6 out of 11 trials. Failures were primarily due to hardware or network issues, not the VADER logic.
  • User Study: VADER significantly outperformed the control (a robot that didn't ask for help) on the metric of whether users felt they could "successfully help the robot" ($p = .003$).
  • Accuracy: The system achieved a 90% success rate in self-variant classification and a 98% success rate in detecting table clutter.
Quantitative results from the HRI study showing VADER's performance against a control.

Limitations and Future Work

  • Latency: The current system is slow. In the user study, VADER episodes took 3-4x longer than the control, with 65% of that time spent waiting for cloud-based LLM responses.
  • Social Norms: The current "nearest person" logic for seeking help is rudimentary. Future versions should respect social context (e.g., not interrupting a conversation).
  • Dialog: Help requests are currently one-way. Implementing multi-turn dialog would allow humans to ask for more context or provide better assistance.
  • Helper Selection: Future work could involve the LMP reasoning about which specific agent (human vs. specific robot type) is best suited for a particular help request.

Summary updated

Abstract

Robots today can exploit the rich world knowledge of large language models to chain simple behavioral skills into long-horizon tasks. However, robots often get interrupted during long-horizon tasks due to primitive skill failures and dynamic environments. We propose VADER, a plan, execute, detect framework with seeking help as a new skill that enables robots to recover and complete long-horizon tasks with the help of humans or other robots. VADER leverages visual question answering (VQA) modules to detect visual affordances and recognize execution errors. It then generates prompts for a language model planner (LMP) which decides when to seek help from another robot or human to recover from errors in long-horizon task execution. We show the effectiveness of VADER with two long-horizon robotic tasks. Our pilot study showed that VADER is capable of performing complex long-horizon tasks by asking for help from another robot to clear a table. Our user study showed that VADER is capable of performing complex long-horizon tasks by asking for help from a human to clear a path. We gathered feedback from people (N=19) about the performance of the VADER performance vs. a robot that did not ask for help. https://google-vader.github.io/
BibTeX
@misc{ahn2024vader,
  author = {Michael Ahn and Montserrat Gonzalez Arenas and Matthew Bennice and Noah Brown and Christine Chan and Byron David and Anthony Francis and Gavin Gonzalez and Rainer Hessmer and Tomas Jackson and Nikhil J Joshi and Daniel Lam and Tsang-Wei Edward Lee and Alex Luong and Sharath Maddineni and Harsh Patel and Jodilyn Peralta and Jornell Quiambao and Diego Reyes and Rosario M Jauregui Ruano and Dorsa Sadigh and Pannag Sanketi and Leila Takayama and Pavel Vodenski and Fei Xia},
  title = {{VADER: Visual Affordance Detection and Error Recovery for Multi Robot Human Collaboration}},
  year = {2024},
  eprint = {2405.16021},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.16021},
}
Figure