LLM-RSPF: Large Language Model-Based Robotic System Planning Framework for Domain Specific Use-Cases

Chandan Kumar Singh, Devesh Kumar, Vipul Sanap, Rajesh SinhaWACV 2025

TL;DR

This paper introduces LLM-RSPF, a novel framework for enabling Large Language Models (LLMs) to perform complex robotic task planning in domain-specific use-cases (DSUs). By combining a structured robotic system ontology with a new "Chain of Hierarchical Thought" (CoHT) prompting technique, LLM-RSPF significantly improves planning accuracy and robustness, addressing limitations of direct LLM application in robotics.

What Problem Does This Paper Address?

The increasing capabilities of LLMs have opened doors for revolutionizing human-robot interaction and task execution. However, directly applying off-the-shelf LLMs to robotic task planning for industrial-grade, domain-specific use-cases (DSUs) falls short. This is due to several challenges: LLMs often exhibit inconsistent responses, lack specific domain knowledge, and struggle with the physical understanding of robot-world interactions. Furthermore, DSUs demand high levels of accuracy, robustness, and throughput, which current LLM applications often fail to meet. This paper aims to bridge this gap by proposing a framework that effectively contextualizes LLMs for these demanding robotic applications.

Methods

The core of the LLM-RSPF framework lies in two novel components:

  • Robotic System Ontology: This structured and modular ontology organizes the components of a robotic system and the DSU. It includes modules for:
  • Use-case: Defines the problem statement and DSU description.
  • Embodiment: Describes the agents (robots, behaviors, tools, sensors) involved, detailing their capabilities, interfaces, states, and experiences.
  • Workspace: Defines the physical environment, including mobility spaces, object locations, and arrangements.
  • Objective: Sets the task objectives and domain rules that the generated plan must adhere to, along with performance metrics.
  • Relation: Captures relational mappings between components.
  • Experience: Stores past success and failure scenarios to inform future planning.
  • Chain of Hierarchical Thought (CoHT) Prompting: Inspired by Bloom's taxonomy and building upon Chain of Thought (CoT), CoHT employs a hierarchical prompting strategy. Instead of linear reasoning steps, it moves from abstract thinking to lower-level, detailed reasoning. This approach helps LLMs better handle ambiguous instructions and complex DSU reasoning by first abstracting the task and then extending to object-level or behavior-level details. It also allows for explicit highlighting of hard-bound rules to prevent learning stagnation.

The framework also incorporates an empirical methodology for generating LLM-tuning datasets, considering unit size, diversity, and uncertainty to ensure desired contextualization.

Datasets

The paper describes the creation of a dataset for training, validation, and testing the LLM-RSPF. This dataset is structured based on the framework's ontology and includes:

  • Training, Validation, and Testing Splits: The dataset is divided in a 1:0.5:2 ratio.
  • Dataset Unit Size Calculation: The size is estimated based on instruction classifications and robotic skills, incorporating factors for diversity and uncertainty.
  • Human-Oracle Annotations: The dataset of 140 instructions was created by 14 human oracles briefed on the framework.

The paper does not introduce a new benchmark dataset but rather describes the methodology for creating a dataset tailored to their proposed framework for a specific DSU.

Key Insights and Ideas

  • LLMs need structure for complex robotics: Direct application of LLMs to robotic task planning is insufficient for DSUs; a structured framework is essential.
  • Ontology is key to contextualization: A well-defined robotic system ontology provides the necessary context for LLMs to understand the DSU's components and rules.
  • Hierarchical reasoning improves LLM performance: CoHT's move from abstract to detailed reasoning is more effective than linear CoT for complex robotic tasks, especially with less capable LLMs.
  • Experience matters: Storing and recalling past success and failure experiences is crucial for improving LLM planning robustness and reducing latency.
  • Dataset design is critical: The methodology for creating diverse and uncertainty-aware datasets is vital for effective LLM tuning.

Key Results

The LLM-RSPF framework, when applied to a retail order-fulfillment DSU, demonstrated exceptional performance:

  • Superior Accuracy: LLM-RSPF achieved 100% plan accuracy, significantly outperforming existing methods like MCRM (54.54%) and ProgPrompt (45.45%) in plan accuracy and classification.
  • Robustness: The framework showed high classification accuracy (0.92) and recall (0.87).
  • CoHT Efficacy: CoHT outperformed CoT and ToT in plan accuracy for both GPT3.5 and GPT4-Turbo LLMs, with a particularly significant margin for GPT4-Turbo.
  • Scalability: The framework demonstrated scalability by successfully handling an extended DSU with multiple agents and increased capabilities, achieving a plan accuracy of 0.91 with GPT4-Turbo.
  • Cost-Effectiveness: The paper emphasizes that LLM-RSPF aims to achieve high performance using LLM-prompting for cost reduction, contrasting with expensive fine-tuning or high compute requirements.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that even with LLM-RSPF, there are specific instructions where all compared methods failed to classify correctly (e.g., an "invalid" case where the LLM failed to re-check object counts, and an "ADR" case where it misidentified a pick location).
  • While the paper discusses the cost-effectiveness of prompting, a detailed comparative cost analysis of the LLM-RSPF itself versus other methods is not fully elaborated beyond the LLM cost comparison.
  • Future Work:
    • The authors suggest exploring the inclusion of knowledge of the 3D world through textual descriptions.
  • Integrating a general 3D perception model is also proposed as a future direction.

Summary updated

BibTeX
@inproceedings{singh2025llm,
  author = {Chandan Kumar Singh and Devesh Kumar and Vipul Sanap and Rajesh Sinha},
  title = {{LLM-RSPF: Large Language Model-Based Robotic System Planning Framework for Domain Specific Use-Cases}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure