Large Language Models as Planning Domain Generators

James T. Oswald, Kavitha Srinivas, Harsha Kokel, Junkyu Lee, Michael Katz, Shirin Sohrabi

Abstract

This research investigates the capacity of Large Language Models (LLMs) to automate the "knowledge engineering" bottleneck in AI planning by translating natural language descriptions into structured Planning Domain Description Language (PDDL). By introducing a rigorous automated evaluation framework based on functional plan equivalence, the study benchmarks several models across classic and novel domains. The findings reveal that while high-parameter models like LLaMA-2-70b achieve high syntactic accuracy, the logical reconstruction of domain actions remains a significant challenge, highlighting a persistent gap between linguistic fluency and symbolic reasoning.

Introduction: The Representation Gap

In the field of AI planning, symbolic methods such as PDDL are highly effective for solving complex logic puzzles and logistics problems. However, these methods suffer from a significant "representation gap." Creating domain models requires specialized technical expertise to translate real-world requirements into rigid, error-prone code. This manual process creates a bottleneck that limits the scalability of automated planning systems.

While Large Language Models (LLMs) have demonstrated remarkable capabilities in generating creative text and general-purpose programming code, their ability to generate highly structured, logically sound symbolic representations for planning is less understood. Previous evaluations in this space often relied on subjective human expertise. This paper addresses this gap by asking: Can LLMs accurately reconstruct planning domains from text, and how can we measure their success without a human in the loop?

Methodology

To answer these questions, the authors propose an action-by-action reconstruction task. Rather than asking an LLM to generate an entire world model at once, the task provides the model with a list of "Allowed Predicates" (the building blocks of the domain) and a natural language description of a specific action (e.g., "how to fly an airplane").

Prompting Strategies

The study tested three distinct classes of natural language descriptions to observe how context affects performance:

  1. Base: Minimal information including the action name, parameters, and a one-line summary.
  2. Flipped: The base information supplemented with explicit mentions of what changes (deleted preconditions).
  3. Random: The base information plus random predicates to test the model's resilience to "noise."

Automated Evaluation

A key innovation of this work is the move beyond simple string matching. The authors introduced Heuristic Domain Equivalence, a metric that checks if the generated PDDL is functionally viable. Using a plan validator, the framework checks if plans derived from the original "ground truth" domain still function within the LLM-generated domain, and vice versa. Outputs are categorized into a hierarchy: Syntax Errors, Semantic Errors (e.g., incorrect argument counts), Different Domains (valid syntax but incorrect logic), and Equivalent Domains (the target goal).

Datasets and Experimental Setup

The study evaluated seven LLMs, including StarCoder and the LLaMA-2 family (7b, 13b, and 70b versions), across nine PDDL domains. These included classic benchmarks like Blocksworld, Gripper, and Logistics, as well as novel domains created specifically to avoid training data leakage, such as Heavy and Trackbuilding.

Key Insights and Results

The Impact of Model Size and Tuning

Model size proved to be a critical factor in performance. LLaMA-2-70b emerged as the top performer, successfully reconstructing 24.84% of domains to be heuristically equivalent to the ground truth. It also demonstrated high reliability in syntax, with an error rate of only 0.31%, compared to over 21% for the 7b-chat model.

Interestingly, "Chat" versions of models (fine-tuned with RLHF) often performed worse than their base counterparts. This suggests that chat tuning, which prioritizes conversational fluency, may inadvertently degrade a model's ability to adhere to the rigid structural requirements of symbolic code.

The "ARE" Trap

The authors identified that Action Reconstruction Error (ARE)—a simple count of predicate accuracy—is a misleading metric. A model might have a low error count but produce a logically useless domain, or conversely, a high error count while producing a functionally equivalent one. This reinforces the necessity of functional evaluation over lexical comparison.

Logic vs. Syntax

While high-end models have largely "solved" PDDL syntax, the logic remains a hurdle. Even when models produced valid PDDL that appeared correct, the majority (56-74%) resulted in a "Different Domain" where original plans would fail. This indicates that LLMs still struggle to capture the precise causal relationships required for symbolic planning.

Limitations and Future Work

The current approach assumes the LLM is provided with a pre-defined list of valid predicates. A truly autonomous system would need to generate these predicates from raw narratives. Additionally, because the study evaluates actions in isolation, it does not account for how errors might compound across an entire domain.

Future research should explore iterative refinement. By integrating tools like VAL (a plan validator) into the loop, LLMs could receive feedback on their errors and self-correct, mimicking the iterative workflow of a human knowledge engineer.

Summary updated

Abstract

Developing domain models is one of the few remaining places that require manual human labor in AI planning. Thus, in order to make planning more accessible, it is desirable to automate the process of domain model generation. To this end, we investigate if large language models (LLMs) can be used to generate planning domain models from simple textual descriptions. Specifically, we introduce a framework for automated evaluation of LLM-generated domains by comparing the sets of plans for domain instances. Finally, we perform an empirical analysis of 7 large language models, including coding and chat models across 9 different planning domains, and under three classes of natural language domain descriptions. Our results indicate that LLMs, particularly those with high parameter counts, exhibit a moderate level of proficiency in generating correct planning domains from natural language descriptions. Our code is available at https://github.com/IBM/NL2PDDL.
BibTeX
@inproceedings{oswald2024large,
  author = {James T. Oswald and Kavitha Srinivas and Harsha Kokel and Junkyu Lee and Michael Katz and Shirin Sohrabi},
  title = {{Large Language Models as Planning Domain Generators}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30491},
}
Figure