Building and Using a Planning Ontology from Past Data for Performance Efficiency

Bharath Muppasani, Vishal Pallagani, Biplav Srivastava, Raghava MutharajuPLATO@ICAPS 2023 · OpenReview

Abstract

Automated planning research has generated vast amounts of data through competitions like the International Planning Competition (IPC), yet much of this knowledge remains unstructured and underutilized. This paper presents a domain-independent planning ontology designed to formalize IPC data into a queryable knowledge graph. By leveraging this ontology, we demonstrate the automated extraction of macro-operators—sequences of actions that serve as planning shortcuts. We evaluate the utility of these macros by integrating them into the prompts of Plansformer, a transformer-based planner. Results show significant performance gains in the Blocksworld domain, though challenges remain in more complex domains where additional prompt information can introduce noise.

Introduction

Automated planning is a cornerstone of AI, tasking agents with finding a sequence of actions to reach a goal. Over the years, the community has produced a mountain of data through the International Planning Competition (IPC), including various domains, planners, and performance metrics. However, this data is largely "dark"—it is unstructured and difficult to query. There is a lack of a unified, formal framework to represent this knowledge, making it hard to build "meta-planners" that learn from past successes and failures.

Methods

The authors propose a structured Planning Ontology built using standard Semantic Web tools (Protege, OWL, and RDF). The ontology defines 19 distinct classes (such as Domain, Action, Planner, and Cost) and 24 object properties to link them. The ontology categorizes information into four main buckets: the problem domain, the specific problem instance, the generated plan/cost, and the historical performance of the planner.

The proposed Planning Ontology structures relationships between domains, problems, actions, and planner performance.

A key application of this ontology is the automated extraction of macro-operators. Using SPARQL queries against the knowledge graph, the researchers identify frequent sequences of actions. To ensure these macros are valid, they analyze preconditions and effects to verify logical consistency. These macros are then "infused" into the prompts of Plansformer, a Large Language Model (LLM) fine-tuned for planning tasks, as a form of in-context learning.

Datasets and Evaluation

The ontology was populated using historical data from the IPC (1998, 2000, and 2002). The evaluation focused on three classic planning domains:

  • Blocksworld (bw): Moving blocks to reach a target configuration.
  • Driverlog (dl): Logistics involving trucks, drivers, and packages.
  • Grippers (gr): Robots moving balls between rooms.

Key Results

The researchers compared the performance of Plansformer with and without the extracted macros. In the Blocksworld domain, adding macros improved the success rate from 90.04% to 94.08%, and the percentage of optimal plans rose from 88.44% to 92.36%.

Performance comparison of Plansformer with and without macro-operators across three planning domains.

However, performance declined in the driverlog and grippers domains when macros were added. For example, driverlog success dropped from 76.56% to 40.08%. The authors observed that in more complex domains, simply appending macros to the prompt might have introduced too much noise or exceeded the effective context handling of the LLM, leading to more failed or incomplete plans.

Conclusion

This research demonstrates that planning data is most valuable when structured as a graph rather than a simple list of results. While the ontology successfully extracted macros that boosted performance in simpler domains, the method of "infusing" them into LLMs needs refinement to avoid confusing models in complex scenarios. Future work will focus on expanding the ontology to modern IPC domains and developing more sophisticated retrieval methods to ensure only the most relevant macros are utilized.

Summary updated

BibTeX
@inproceedings{muppasani2023building,
  author = {Bharath Muppasani and Vishal Pallagani and Biplav Srivastava and Raghava Mutharaju},
  title = {{Building and Using a Planning Ontology from Past Data for Performance Efficiency}},
  booktitle = {PLATO@ICAPS},
  year = {2023},
  url = {https://openreview.net/forum?id=mFOVag2Jel},
}
Figure