TL;DR
This paper introduces a novel "Planning Augmented Semantic Parsing" method that uses symbolic planning to improve how large language models (LLMs) understand and generate executable plans from instructional texts, especially in low-data scenarios. By integrating a symbolic planner, the system significantly reduces errors in generated plans, making them more reliable for real-world execution.
What Problem Does This Paper Address?
Large language models (LLMs) have shown impressive capabilities in understanding and generating human-like text. However, when it comes to instructional texts (like recipes or assembly guides), LLMs often struggle with:
- Long-context dependencies: Instructions can be lengthy, with actions and requirements spread across many sentences.
- Ambiguity and domain-specific language: The language used can be informal, context-dependent, and specific to a particular domain (e.g., cooking terms).
- Executable plans: Simply generating text that looks like instructions isn't enough. The output needs to be a sequence of actions that can actually be executed, respecting logical preconditions (e.g., you can't bake something if the oven isn't preheated).
Existing LLM approaches, while powerful, often fail to guarantee the logical consistency and executability of these generated plans, especially when trained on limited data (few-shot learning). This paper aims to bridge this gap by combining the strengths of LLMs with the rigor of symbolic planning.
Methods
The core innovation is a Planning Augmented Semantic Parsing approach. Here's how it works:
- LLM-based Initial Parsing: The system uses a powerful LLM (specifically, OpenAI's Davinci Codex) to generate initial semantic parses (action sequences) from instructional texts. This is done in a few-shot setting, where the LLM is prompted with a few examples of instructions and their corresponding parses.
- Planning-Augmented Decoding: Instead of just accepting the LLM's first output, the system employs a multi-stage decoding process:
- Sampling Completions: The LLM generates multiple candidate plan completions for each instruction.
- Ranking: These candidate plans are ranked using a novel scoring function. This score combines the LLM's confidence (sequence probability) with a "planning score." The planning score penalizes plans based on the number of precondition errors (actions that can't be executed due to unmet conditions) and syntax errors, while also considering the number of additional steps needed to fix these errors.
- Correcting Plans: The highest-ranked plan is then passed to a symbolic planning module. This module uses domain-specific planning information (defining objects, states, and actions) to identify and insert missing actions that satisfy unmet preconditions, thereby correcting errors and ensuring executability.
The key novelty lies in using the symbolic planner not just to generate plans from scratch, but to refine and correct plans initially generated by an LLM, leveraging the LLM's ability to handle natural language and the planner's guarantee of logical consistency.
Datasets
The authors evaluate their method on two existing recipe semantic parsing datasets:
- (Tasse and Smith, 2008): Contains 260 recipes with semantic parses in the Minimal Instruction Language for the Kitchen (MILK) syntax. This dataset covers a wide variety of cuisines and ingredients.
- (Bollini et al., 2013): Contains 60 recipes, primarily baking, designed for execution by a cooking robot. This dataset also includes planning domain definitions.
The paper uses leave-one-out cross-validation for evaluation.
Key Insights and Ideas
- Neuro-Symbolic Synergy: The paper strongly advocates for combining deep learning (LLMs) with symbolic reasoning (planning). LLMs excel at understanding nuanced language and generating plausible sequences, while symbolic planners provide the logical guarantees needed for executability.
- Planning for Correction, Not Just Generation: The most impactful insight is using planning as a correction mechanism for LLM outputs, rather than solely relying on it for initial plan generation. This leverages the LLM's strengths while mitigating its weaknesses.
- The Trade-off Between Executability and Semantic Correctness: The authors highlight that simply making plans executable (e.g., by adding many generic actions) can sometimes reduce their semantic alignment with the original instructions (measured by LCS). Their method aims to balance these two crucial aspects.
- Domain Knowledge is Key: The effectiveness of the planning component relies on having planning domain definitions. While this requires effort, the paper suggests LLMs might even be able to assist in generating these definitions.
Key Results
- Reduced Precondition Errors: The "Rank + Plan" method significantly reduces precondition errors (PE) compared to baselines like "No Rank" and "Rank (PPL)". For the Bollini dataset with 5 examples, PE drops from 0.277 to 0.033. For Tasse, PE drops from 0.805 to 0.000.
- Improved F1 Score: The F1 score, which measures the accuracy of individual plan steps, also improves with the "Rank + Plan" method, indicating more accurate step generation.
- Maintained LCS: The Longest Common Subsequence (LCS) metric, which measures overall plan similarity to ground truth, remains largely stable or only slightly decreases, demonstrating that the plans still closely follow the original instructions while becoming more executable.
- Few-Shot Performance: The method shows strong performance even with limited training examples (few-shot learning), which is crucial given the cost of annotating long instructional texts.
Limitations and Future Work
- Reliance on Planning Domain Information: The approach requires explicit planning domain definitions for each task. Creating these definitions can be labor-intensive, requiring programming and domain knowledge.
- Potential for LLM-Generated Domain Definitions: The authors note that LLMs like Codex can generate planning domain definitions in various formats, suggesting a future direction for automating this process.
- Generalization to Other Domains: While evaluated on recipes, future work could explore applying this method to other domains like virtual home environments or robotic tasks (e.g., ALFRED, VirtualHome).
- Balancing LCS and F1: The paper acknowledges a potential trade-off where increasing executability might slightly decrease LCS. Future work could further refine this balance.
Summary updated