Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliability

Md Sadman Sakib, Yu Sun

Large Language Models (LLMs) are surprisingly good at "common sense" reasoning, making them tempting tools for robotic task planning. If you ask GPT-4 how to make a salad, it can give you a decent step-by-step guide. However, robots require more than just "decent" instructions—they need precise, executable, and optimal plans. LLMs are notorious for "hallucinations" (generating plausible-sounding but incorrect steps) and often suggest high-cost actions that a specific robot might struggle to perform.

A new paper from researchers at the University of South Florida introduces a framework that moves beyond single-shot LLM planning. By generating multiple plans and "consolidating" them into a verified graph, they’ve created a system that is significantly more reliable and efficient than standard LLM approaches.

TL;DR

This paper presents a framework that uses GPT-4 to generate multiple potential task trees for a robot, merges them into a unified network, and filters out errors using a curated knowledge base (FOON). By selecting the most cost-effective path through this network and converting it into PDDL code, the system produces highly accurate, robot-executable plans that outperform standard LLM-based methods.

What Problem Does This Paper Address?

Traditional robotic planning usually relies on "closed-world" knowledge bases. These systems work perfectly until they encounter something new—like a recipe they haven't seen before—at which point they fail completely. LLMs offer an "open-world" alternative, but they are probabilistic and unpredictable. An LLM might suggest "un-chopping" a vegetable or propose a sequence of actions that is physically exhausting or impossible for a specific robot arm.

The authors tackle three main gaps:

  1. Unreliability: LLMs can produce logically flawed steps.
  2. Sub-optimality: LLMs don't naturally account for the "cost" of an action (e.g., is it easier for this robot to pour or to scoop?).
  3. Execution Gap: High-level natural language instructions need to be translated into low-level, machine-executable code like PDDL (Planning Domain Definition Language).

Methods

The authors propose a multi-stage pipeline:

  1. Multi-Tree Generation: Instead of asking for one plan, the system uses GPT-4 to generate five distinct "task trees" for a given instruction (e.g., "Make a noodle soup").
  2. Graph Amalgamation: These trees are merged into a single "Unified Network." This allows the system to see multiple ways to achieve the same goal.
  3. Verification via FOON: The system checks the merged graph against the Functional Object-Oriented Network (FOON), a curated knowledge base of valid state transitions. If the LLM suggests an impossible transition (like "chopped $\rightarrow$ whole"), it is discarded.
  4. Cost-Effective Search: The system assigns "costs" to different motions (e.g., scooping might cost 0.4 while pouring costs 0.1 for a UR5 robot). It then uses a Depth-First Search to find the path with the lowest total execution cost.
  5. PDDL Conversion: Finally, GPT-4 converts the optimal high-level functional units into PDDL domain and problem files, which are then solved by a classical planner (Fast-Downward) to generate low-level robot commands.

Datasets

  • FOON (Functional Object-Oriented Network): A core component of the pipeline, containing 150 manually annotated and verified cooking recipes used for grounding and error-checking.
  • Recipe1M+: A massive dataset of over a million recipes used to curate a test set of 400 recipes across four categories: Salad, Drink, Muffin, and Omelette.

Key Insights and Ideas

  • The "Wisdom of the LLM Crowd": By generating multiple plans, the system can bypass a "hallucination" in one tree by finding a correct path in another.
  • Knowledge-Grounded Filtering: Using a small, high-quality dataset (FOON) to "police" a massive, noisy model (GPT-4) provides the best of both worlds: the creativity of the LLM and the reliability of a symbolic system.
  • Robot-Specific Optimization: By incorporating execution costs, the planner doesn't just find a way to do the task; it finds the best way for that specific robot's hardware configuration.
  • Human-in-the-Loop Visualization: The authors developed "progress lines" that allow humans to easily visualize the plan and make manual corrections before the robot starts moving.

Key Results

The researchers compared their approach against several baselines, including fine-tuned GPT-3 and standard prompt-engineered GPT-4.

  • Superior Accuracy: The "Ours (with FOON)" model consistently achieved the highest accuracy across all dish categories, significantly outperforming FOON-only search and standalone LLM prompts.
  • Efficiency Gains: By merging just five task trees, the system found a more cost-effective plan in 25% of cases compared to the first plan generated by the LLM.
  • Generalizability: While tested on cooking, the authors demonstrated the system's ability to generalize to other domains, such as packing a suitcase, with minimal adjustments.

Limitations and Future Work

  • Search Complexity: As the unified network grows (by adding more LLM trees or external knowledge), the time required to retrieve the optimal tree increases. The authors suggest using heuristic searches like RRT* to mitigate this.
  • Lack of Real-Time Feedback: The current system is a "pre-planner." It doesn't yet use real-time visual feedback to recover if a robot drops an ingredient mid-execution.
  • Future Directions: The authors plan to integrate the Tree-of-Thoughts (ToT) approach to allow the LLM to evaluate its own reasoning at each step and to incorporate real-time environmental data to handle execution failures autonomously.

Summary updated

Abstract

The inherent probabilistic nature of Large Language Models (LLMs) introduces an element of unpredictability, raising concerns about potential discrepancies in their output. This paper introduces an innovative approach aims to generate correct and optimal robotic task plans for diverse real-world demands and scenarios. LLMs have been used to generate task plans, but they are unreliable and may contain wrong, questionable, or high-cost steps. The proposed approach uses LLM to generate a number of task plans as trees and amalgamates them into a graph by removing questionable paths. Then an optimal task tree can be retrieved to circumvent questionable and high-cost nodes, thereby improving planning accuracy and execution efficiency. The approach is further improved by incorporating a large knowledge network. Leveraging GPT-4 further, the high-level task plan is converted into a low-level Planning Domain Definition Language (PDDL) plan executable by a robot. Evaluation results highlight the superior accuracy and efficiency of our approach compared to previous methodologies in the field of task planning.
BibTeX
@article{sakib2024consolidating,
  author = {Md Sadman Sakib and Yu Sun},
  title = {{Consolidating Trees of Robotic Plans Generated Using Large Language Models to Improve Reliability}},
  journal = {International Journal of Artificial Intelligence and Robotics Research},
  year = {2024},
  doi = {10.1142/s2972335324500029},
}
Figure