Towards Robots That Know When They Need Help: Affordance-Based Uncertainty for Large Language Model Planners

James F. Mullen Jr., Dinesh Manocha

Abstract

Large language models (LLMs) showcase many desirable traits for intelligent and helpful robots. However, they are also known to hallucinate predictions. This issue is exacerbated in robotics where LLM hallucinations may result in robots confidently executing plans that are contrary to user goals, relying more frequently on human assistance, or preventing the robot from asking for help at all. In this work, we present LBAP (Likelihood-Based Alignment for Planners), a novel approach for utilizing off-the-shelf LLMs alongside a novel Action feasibility metric (A-Feasibility) in robotic planners that minimize harmful hallucinations and human intervention. A-Feasibility measures whether a given action is possible and safe in the provided scene, helping to mitigate hallucinations and better align the LLM's confidence with the probability of success. Through experiments in simulation and the real world, we show that LBAP significantly increases success rate and decreases the amount of human intervention required relative to prior art.

Introduction

While Large Language Models (LLMs) like GPT-4 can generate complex plans from natural language instructions, they suffer from a lack of "grounding." An LLM might confidently suggest "pick up the coffee cup" even if no cup exists in the room, or "put the metal bowl in the microwave," which is a safety hazard. In robotics, this leads to a failure in uncertainty alignment: ideally, a robot should execute a task if it’s certain and ask for help if it’s confused. Current state-of-the-art methods rely on the LLM's internal "confidence" (next-token probabilities), but these values are often poorly calibrated for physical environments.

The Challenge of LLM Hallucinations

The core problem is that LLMs are often "confidently wrong." An LLM might give a hallucinated action a high probability simply because it sounds linguistically plausible in the context of the prompt. This prevents the robot from "knowing what it doesn't know," leading to either dangerous autonomous actions or unnecessary interruptions for human help.

LBAP: Grounding LLMs through Bayesian Inference

The researchers introduce LBAP, which treats the planning problem as a Bayesian update. Instead of just trusting the LLM's initial guess, they refine it using three components that define the A-Feasibility of an action:

  1. The Prior ($P(y_i)$): The raw probability assigned by the LLM to a candidate action in a Multiple Choice Question Answering (MCQA) format.
  2. Scene Grounding ($P(S|y_i)$): A check to see if the objects required for the action actually exist in the environment, using textual descriptions or vision models like ViLD or CLIP.
  3. World Knowledge ($P(W|y_i)$): A separate LLM prompt that evaluates if an action is safe and possible (e.g., "Is it safe to put this in the microwave?").
The LBAP pipeline: Bayesian inference corrects a hallucinated "coffee cup" action by grounding the LLM's initial probabilities in scene context and world knowledge.

By multiplying these terms ($P(y_i|S,W) \propto P(y_i)P(S|y_i)P(W|y_i)$), the robot generates a posterior probability. If the resulting "prediction set" of likely actions contains more than one option (or zero), the robot asks the human for help. This structured approach makes the system more explainable, as users can see exactly why a robot rejected an action (e.g., "Object not found" vs. "Unsafe action").

Experimental Evaluation

LBAP was evaluated using two primary benchmarks:

  • Simulation (Tabletop Rearrangement): A PyBullet environment where a robot arm moves blocks and bowls, featuring injected ambiguities regarding attributes (color/shape), numbers, and spatial relations.
  • Real-World (Mobile Manipulator): A ClearPath TurtleBot operating in a kitchen environment, handling complex scenarios involving recycling, composting, and appliance safety.

Key Results

  • Simulation: LBAP reduced the human help rate by 70% (from 30.1% to 8.9%) at a 60% success rate compared to the KnowNo baseline.
  • Real-World: In kitchen tasks, LBAP achieved a 70% success rate while requiring help only 42% of the time, compared to 63% for the baseline—a 33% improvement in autonomy.
  • Scaling Benefits: Interestingly, the gap between LBAP and prior methods widened with more powerful models like GPT-4. This suggests that as LLMs get smarter, the need for structured Bayesian grounding becomes even more critical to harness that intelligence safely.

Conclusion

LBAP demonstrates that separating planning from feasibility and safety checks allows robots to leverage the intelligence of LLMs while remaining grounded in physical reality. By aligning robot confidence with actual environmental constraints, LBAP significantly reduces the need for human intervention. Future work will explore more complex, long-horizon tasks and tighter integration with real-time, dynamic perception systems.

Summary updated

Abstract

Large language models (LLMs) showcase many desirable traits for intelligent and helpful robots. However, they are also known to hallucinate predictions. This issue is exacerbated in robotics where LLM hallucinations may result in robots confidently executing plans that are contrary to user goals, relying more frequently on human assistance, or preventing the robot from asking for help at all. In this work, we present LAP, a novel approach for utilizing off-the-shelf LLMs, alongside a novel Action feasibility metric, in robotic Planners that minimize harmful hallucinations and human intervention. Our key finding is that calculating and leveraging a new metric, which we call A-Feasibility, a measure of whether a given action is possible and safe in the provided scene, helps to mitigate hallucinations in LLM predictions and better align the LLM's confidence measure with the probability of success. We specifically propose an A-Feasibility metric which both combines scene context and prompting a LLM to determine if a given action is possible and safe in the scene, using the LLM's response to compute the score. Through experiments in both simulation and the real world on tasks with a variety of ambiguities, we show that LAP significantly increases success rate and decreases the amount of human intervention required relative to prior art. For example, in our real-world testing paradigm, LAP decreases the human help rate of previous methods by over 33% at a success rate of 70%.
BibTeX
@inproceedings{jr2024towards,
  author = {James F. Mullen Jr. and Dinesh Manocha},
  title = {{Towards Robots That Know When They Need Help: Affordance-Based Uncertainty for Large Language Model Planners}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
}
Figure