Task and Motion Planning with Large Language Models for Object Rearrangement

Yan Ding, Xiaohan Zhang, Chris Paxton, Shiqi Zhang

This paper, titled "Task and Motion Planning with Large Language Models for Object Rearrangement," introduces LLM-GROP, a framework that bridges the gap between the high-level "common sense" of Large Language Models (LLMs) and the low-level physical constraints of robot motion planning.

TL;DR

LLM-GROP enables robots to perform complex object rearrangement tasks (like setting a dinner table) from vague natural language commands. It uses LLMs to determine semantically correct object placements, filters those ideas through symbolic logic to prevent "hallucinations," and optimizes the robot's physical path to ensure the plan is actually executable in the real world.

What Problem Does This Paper Address?

While robots are increasingly capable of "pick and place" actions, they struggle with underspecified tasks. If you tell a robot to "set the table," it doesn't inherently know that a fork should be on the left of the plate or that a knife goes on the right.

Existing solutions usually fall into two traps:

  1. Hard-coded rules: These are brittle and don't generalize to new objects or environments.
  2. Pure LLM planning: LLMs (like GPT-3) have great "common sense" but no "physical sense." An LLM might suggest a perfect table layout but fail to realize that a chair is blocking the robot's path or that two objects are overlapping in 3D space.

The challenge is grounding the abstract knowledge of an LLM into a Task and Motion Plan (TAMP) that accounts for reachability, collisions, and efficiency.

Methods

The authors propose a multi-stage pipeline called LLM-GROP (Large Language Model for Grounded RObot Task and Motion Planning):

  1. Symbolic Relationship Generation: The robot uses a "few-shot" prompt to ask an LLM for the typical layout of objects (e.g., "Where does the fork go relative to the plate?").
  2. Logic Filtering (ASP): To prevent the LLM from suggesting impossible configurations (e.g., Object A is left of B, and B is left of A), the system uses Answer Set Programming (ASP). This acts as a "logic police" to ensure the symbolic plan is consistent.
  3. Geometric Grounding: A second LLM prompt asks for specific distances (e.g., "How many centimeters to the left?").
  4. Adaptive Sampling: Because LLM-provided coordinates might result in collisions or objects falling off the table, the system treats the LLM's output as a mean for a 2D Gaussian distribution. It then samples valid positions that respect the table's boundaries and object sizes.
  5. TAMP Optimization: Finally, the system uses the GROP (Grounded Robot Task and Motion Planning) algorithm to pick the most efficient sequence of movements, choosing the best side of the table to approach based on obstacles like chairs.

Datasets

The paper does not use a traditional training dataset, as it leverages the zero-shot/few-shot capabilities of pre-trained LLMs (specifically text-davinci-003).

  • Evaluation Tasks: The authors designed 8 distinct table-setting tasks (ranging from 3 to 5 objects).
  • Human Evaluation: They collected 3,200 ratings from five human participants who scored the resulting table arrangements on a scale of 1 to 5 based on aesthetic and functional "common sense."

Key Insights and Ideas

  • LLMs as Heuristic Generators: Instead of letting the LLM control the robot directly, LLM-GROP treats the LLM as a source of "priors" or suggestions that are then refined by classical robotics algorithms.
  • The Necessity of Logic: The authors found that LLMs occasionally produce contradictory spatial instructions. Integrating ASP allows the system to catch these errors before the robot even moves.
  • Efficiency vs. Semantics: A key "aha" moment is that the "best" semantic position for an object might be the "worst" for a robot's battery life or safety. LLM-GROP optimizes for a "utility function" that balances how much a human will like the arrangement versus how hard it is for the robot to achieve it.

Key Results

  • Higher User Satisfaction: LLM-GROP achieved a mean human rating of ~4.1/5, significantly outperforming baselines that used random arrangements or LLMs without motion optimization.
  • Faster Execution: By optimizing the task-motion plan, LLM-GROP completed tasks in roughly 120 seconds, compared to 180–220 seconds for baselines that didn't account for navigation efficiency.
  • Real-World Validation: The system was successfully demonstrated on a physical mobile manipulator (a Segway base with a UR5e arm) avoiding real obstacles and a human bystander while setting a table.

Limitations and Future Work

  • Template Dependency: The system currently relies on specific prompt templates. Future versions could benefit from more flexible, conversational interfaces.
  • Hard-coded Grasping: While the placement is planned, the grasping of objects is currently hard-coded. The authors suggest integrating more advanced manipulation models to handle unknown objects.
  • Static Environments: The current model assumes a mostly static table environment during the placement phase; handling highly dynamic environments (like a busy restaurant) remains a future challenge.

Summary updated

Abstract

Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios. Supplementary materials are available at: https://sites.google.com/view/llm-grop
BibTeX
@inproceedings{ding2023task,
  author = {Yan Ding and Xiaohan Zhang and Chris Paxton and Shiqi Zhang},
  title = {{Task and Motion Planning with Large Language Models for Object Rearrangement}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=YEhGbu3uF2},
}

Loading related content...

Figure