TL;DR
This paper introduces DIGGER, a novel LLM-based approach for open-domain planning that leverages entity-attribute-level domain models to overcome limitations of existing methods. It also presents GOLD, the largest dataset for textual procedures with corresponding PDDL definitions, enabling more robust and generalizable planning.
What Problem Does This Paper Address?
Current Large Language Models (LLMs) for open-domain natural language planning (LONG) struggle with generating non-reusable plans, incomplete intermediate states, and missing steps, hindering real-world applications. While using formal planning domain description languages like PDDL can improve performance, challenges include data scarcity, difficulty in building complete domain models, and NL-PDDL translation issues. This paper aims to address these gaps by creating a comprehensive dataset and a new planning framework that bridges the gap between natural language instructions and formal planning representations.
Methods
DIGGER employs a two-stage approach: training and planning.
Training Stage: 1. COMETaction Model Training: A fine-tuned COMET model is trained on the GOLD dataset to extract coarse-grained plan steps from natural language procedures. 2. Entity-Attribute-Level Action Model Building: This stage leverages ConceptNet to identify entity attributes (physical properties) and uses sentence-BERT to embed these attributes. These embeddings are stored in a vector database for efficient retrieval. LLMs then generate entity-attribute-level action models based on these attributes. 3. Task Clustering: WikiHow tasks are clustered using principal component analysis and k-means clustering to group similar tasks.
Planning Stage: 1. Domain Prediction: For a new task, DIGGER predicts its domain based on the similarity of its embedding to existing domain embeddings. 2. Goal State Extraction: A COMETstate model extracts the goal state from the natural language task description. 3. Action Instantiation: Grounded actions are instantiated using the predicted domain's typed entity list and action schemata. For unseen domains, a global typed entity list is used. A threshold determines if a domain is seen or unseen. 4. Few-Shot Prompting: A few-shot prompt is constructed for LLMs, including similar tasks, instantiated actions, and instructions. 5. PDDL Plan Generation: LLMs generate a PDDL plan for the task using the constructed prompt. 6. Textual Procedure Generation: The PDDL plan is translated back into a natural language textual procedure by an LLM.
Datasets
The paper introduces GOLD, a new dataset designed for LONG. GOLD comprises:
- 103 domains
- 1100 tasks
- 1086 actions
It is built from four source datasets: WikiHow, ALFRED-L, Proc2PDDL, and OPENPI+ (an extension of OPENPI with additional PDDL domain definitions). GOLD is highlighted as the largest dataset for textual procedures with corresponding PDDL definitions.
Key Insights and Ideas
- Entity-Attribute-Level Models are Crucial: Incorporating entity attributes (e.g., "plant-location") significantly enhances the generated plans by revealing implicit physical properties and enabling the identification of hidden states and intermediate steps. This is a key differentiator from previous approaches that often ignore such details.
- Leveraging LLMs for PDDL Generation: DIGGER effectively uses LLMs not only for generating natural language plans but also for translating between natural language and PDDL, and for constructing PDDL domain models. This bridges the gap between symbolic planning and the capabilities of LLMs.
- GOLD Dataset's Importance: The GOLD dataset is presented as a vital resource for benchmarking LONG, providing a large-scale, diverse collection of procedural texts paired with formal PDDL definitions. This enables more comprehensive and realistic evaluation.
- LLMs Tolerate NL-PDDL Errors: The paper suggests that LLMs can tolerate errors in NL-PDDL translation, which, when combined with incomplete PDDL definitions, can boost planning performance.
- Hybrid Approach Outperforms Pure LLMs: DIGGER, a hybrid approach combining LLMs with symbolic planning principles and entity-attribute information, demonstrates superior performance compared to pure LLM-based methods and other recent approaches.
Key Results
- Performance on GOLD: DIGGER outperforms three LLMs (DeepSeek-v3, GPT-4, GPT-40) and two recent approaches (Open-Ground-Planning and PLAN) on the GOLD dataset.
- BLEU-1 Scores:
- On seen domains, DIGGER achieved a BLEU-1 score of 0.408, an increase from 0.385.
- On unseen domains, DIGGER achieved a BLEU-1 score of 0.310, significantly higher than the best LLM's score of 0.089 (GPT-4).
- Human Evaluation: DIGGER showed strong performance in human evaluations, with a win-to-loss ratio (WTLR) of 88.6% and a success rate (SR) of 90.0%.
- Ablation Studies: The effectiveness of DIGGER's components was ranked as COMETaction > Entity-Attributes > Instantiation. COMETaction was found to be crucial for PDDL planning, while Entity-Attributes acted as a double-edged sword, sometimes decreasing scores due to ignored references but crucial for unseen domains. Instantiation was vital for LaJ and human evaluation.
Limitations and Future Work
- Limited LLM Resources: Experiments were conducted using LLMs no larger than BART due to resource constraints.
- Fine-tuning Dependency: The performance is dependent on the fine-tuning process.
- LLM-Generated Augmentations: Augmentations generated by LLMs are considered "silver-standard," necessitating human evaluation.
- LLM Stability: The stability of LLMs is out of scope, and experiments used single runs.
- English Language Focus: The study is limited to the English language, potentially limiting generalizability across cultures and dialects.
- Insufficient Annotated Procedures: The annotated procedures are still considered insufficient.
- Incomplete Entity-Attribute Information: Existing entity-attribute information in OPENPI+ and ConceptNet is not enough.
- Lack of Better Automated Metrics: Better automated evaluation metrics for LONG are needed.
Future work will focus on addressing these limitations, including exploring larger LLMs, developing more robust evaluation metrics, and expanding the dataset with more diverse languages and annotated procedures.
Summary updated