Abstract
Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios.
Introduction
Service robots operating in human-centric environments must often perform tasks that are underspecified. A command like "set the table" implies a wealth of cultural and functional knowledge—such as the relative positioning of cutlery and plates—that is rarely explicitly programmed. While Large Language Models (LLMs) possess significant "commonsense" knowledge regarding these semantic relationships, they lack an inherent understanding of physical constraints, reachability, and geometric feasibility.
This paper introduces LLM-GROP (Large Language Model for Grounded RObot Task and Motion Planning), a framework designed to bridge the gap between high-level semantic reasoning and low-level physical execution. By integrating LLMs with Task and Motion Planning (TAMP) and logic-based filtering, LLM-GROP enables robots to translate vague natural language instructions into executable, human-aligned physical plans.
The Challenge: Bridging Semantics and Physics
Traditional approaches to object rearrangement typically fall into two categories: hard-coded heuristics or end-to-end neural planning. Hard-coded rules are brittle and fail to generalize to new objects or environments. Conversely, while LLMs can suggest ideal layouts, they often "hallucinate" physically impossible configurations, such as overlapping objects or placements in unreachable areas.
The core challenge lies in grounding abstract knowledge. A robot must not only know where an object should go semantically but also how to place it there without colliding with obstacles or violating the laws of physics.
The LLM-GROP Framework
LLM-GROP addresses these challenges through a multi-stage pipeline that refines abstract suggestions into concrete actions:
1. Symbolic Relationship Generation
The process begins with few-shot prompting. The system queries an LLM (such as text-davinci-003) to identify the symbolic relationships between objects. For instance, given the task "set the table," the LLM identifies that a "fork" should be "left of" the "plate."
2. Logic Filtering with Answer Set Programming (ASP)
To ensure the LLM's suggestions are internally consistent, LLM-GROP employs Answer Set Programming (ASP). This symbolic logic layer acts as a validator, catching contradictory instructions (e.g., if the LLM suggests Object A is left of B, and B is left of A) before they reach the motion planner.
3. Geometric Grounding and Adaptive Sampling
Once symbolic consistency is established, a second prompt extracts specific spatial coordinates. However, because LLM-provided coordinates may not account for the specific geometry of the robot's environment, the system treats these outputs as the mean of a 2D Gaussian distribution. LLM-GROP then samples valid positions from this distribution that respect the physical boundaries of the surface and the dimensions of the objects.
4. TAMP Optimization
The final stage involves the GROP algorithm, which optimizes the sequence of robot movements. It evaluates the "utility" of various configurations, balancing semantic correctness (how well the layout matches human expectations) with execution efficiency (path length and obstacle avoidance). This ensures the robot doesn't just place objects correctly, but does so in a way that is physically optimal.
Experimental Evaluation
The authors evaluated LLM-GROP across eight distinct table-setting tasks involving various object counts and environmental obstacles.
Human-Centric Metrics
A critical component of the evaluation was human alignment. Through a study involving 3,200 individual ratings, participants scored the robot's final arrangements on a scale of 1 to 5. LLM-GROP achieved a mean rating of ~4.1/5, significantly outperforming baselines that relied on random placement or LLM planning without geometric grounding.
Operational Efficiency
Beyond aesthetics, LLM-GROP demonstrated superior performance in execution. By optimizing the task-motion plan, the system completed rearrangement tasks in approximately 120 seconds, a marked improvement over the 180–220 seconds required by baselines that did not account for navigation and reachability during the planning phase.
Real-World Implementation
The framework was validated on a physical mobile manipulator consisting of a Segway base and a UR5e arm. In real-world tests, the robot successfully navigated around obstacles and human bystanders to arrange items on a table, proving that the LLM-derived plans are robust enough for dynamic, physical environments.
Discussion and Future Work
While LLM-GROP represents a significant step forward, several avenues for future research remain. The current system relies on predefined prompt templates; moving toward more fluid, conversational interfaces would enhance usability. Additionally, while object placement is planned, the grasping mechanics remain hard-coded. Integrating advanced manipulation models could allow the system to handle a wider variety of novel objects. Finally, extending the framework to handle highly dynamic environments remains a key objective for future iterations.
Conclusion
LLM-GROP demonstrates that Large Language Models can serve as powerful heuristic generators for robotics when properly grounded. By combining the semantic breadth of LLMs with the rigor of symbolic logic and the precision of motion planning, we can create robots that not only understand "what" to do but also "how" to do it in a way that aligns with human common sense.
Summary updated