Abstract
In this work, we introduce SMART-LLM, an innovative framework designed for embodied multi-robot task planning. SMART-LLM harnesses the power of Large Language Models (LLMs) to convert high-level task instructions into a multi-robot task plan. It accomplishes this by executing a series of stages, including task decomposition, coalition formation, and task allocation, all guided by programmatic LLM prompts within the few-shot prompting paradigm. We create a benchmark dataset for validating multi-robot task planning, encompassing four distinct categories of high-level instructions. Our evaluation experiments span both simulation and real-world scenarios, demonstrating promising results for generating efficient multi-robot task plans.
Introduction
Multi-robot task planning (MRTP) is a fundamental challenge in robotics, particularly when dealing with heterogeneous teams—robots with varying physical capabilities and skill sets. Traditional approaches often rely on rigid, hand-coded logic or complex optimization algorithms that struggle with the ambiguity of natural language instructions. For instance, a human might request a robot team to "prepare a meal in a clean kitchen," an instruction that is highly underspecified and requires significant "common sense" reasoning to decompose into actionable steps.
SMART-LLM addresses these challenges by leveraging the reasoning and coding proficiency of Large Language Models. By treating the planning problem as a code-generation task, the framework can effectively bridge the gap between high-level human intent and low-level robot execution.
The SMART-LLM Framework
The core innovation of SMART-LLM is its multi-stage pipeline, which uses "Pythonic" prompting to guide the LLM through the planning process. Instead of standard natural language prompts, the system provides the LLM with structured Python code snippets as context, allowing it to utilize its training on logical constructs and API usage.
The framework operates in four primary stages:
- Task Decomposition: The LLM receives a high-level instruction and a list of available robot skills. It breaks the instruction into a sequence of independent or dependent sub-tasks. For example, "Close the laptop and watch TV" is decomposed into sub-tasks like "Find the Laptop," "Close the Laptop," and "Turn on the TV."
- Coalition Formation: The system analyzes the sub-tasks alongside a dictionary of robot properties (e.g., skills, payload capacity) and environmental constraints (e.g., object mass). The LLM determines if a sub-task requires a single robot or a team (coalition) of robots.
- Task Allocation: The framework assigns specific robots to the identified sub-tasks. It generates a "coalition policy" that prioritizes efficiency and parallelization. This stage utilizes Python's
threadinglogic to represent tasks that can be performed simultaneously. - Task Execution: The final output is executable Python code. This code calls low-level robot APIs to perform the actions in the environment.
Benchmark Dataset and Evaluation
To evaluate the framework, the authors introduced a new benchmark dataset based on the AI2-THOR simulation platform. The dataset contains 36 high-level instructions categorized by complexity:
- Elemental: Single robot, simple task.
- Simple: Multiple objects, but robots possess all necessary skills.
- Compound: Heterogeneous robots requiring specific skill matching.
- Complex: Requires strategic team assignments due to individual robot limitations (e.g., weight limits requiring two robots to lift an heavy object).
The evaluation focused on success rates and Robot Utilization (RU), a metric measuring the degree of parallelization achieved by the planner.
Key Results and Insights
Experiments using state-of-the-art LLMs like GPT-4 and Claude-3-Opus demonstrated the framework's effectiveness. SMART-LLM achieved a 71% success rate on "Complex" tasks, significantly outperforming baseline methods that relied on random allocation or simpler rule-based logic, which often failed to complete more than 14% of such tasks.
A key insight from the study was the importance of Pythonic prompting. Structuring the prompts as code allowed the LLM to handle multi-agent coordination more reliably than natural language alone. Furthermore, ablation studies showed that including natural language comments within the code prompts acted as a "chain-of-thought" mechanism, improving the LLM's reasoning about task dependencies and robot assignments.
The framework also demonstrated high efficiency, with Robot Utilization scores frequently exceeding 0.90. This indicates that the LLM successfully identified opportunities for parallel execution, minimizing idle time for the robot team.
Conclusion and Future Work
SMART-LLM provides a robust method for translating vague human instructions into coordinated multi-robot plans. By leveraging the "common sense" reasoning of LLMs through structured code generation, it handles heterogeneous teams and complex task constraints effectively.
Current limitations include the constraints of the LLM's context window and the "open-loop" nature of the planning—the system generates a complete plan before execution. Future work will focus on dynamic task allocation, incorporating real-time sensor feedback to allow the system to re-plan and adapt to unexpected changes in the environment.
Summary updated