PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

Arda Uzunoglu, Gözde Gül Sahin, Abdulfattah SafaACL 2024 · DOI

Introduction

Large Language Models (LLMs) are increasingly deployed as autonomous planners. However, standard evaluations often rely on simplified environments like BlocksWorld, which fail to capture the linguistic complexity of real-world tasks. A critical component of effective planning is "implicit knowledge"—the ability to anticipate potential pitfalls (warnings) or best practices (tips) associated with a goal, even without an explicit step-by-step instruction. For instance, when tasked with "Removing Ink Stains from Fabric," a proficient planner should inherently know to blot rather than rub the stain.

Figure 1: Example of a procedural task from wikiHow illustrating the relationship between a goal, steps, and a critical warning.

The PARADISE Dataset

To address this gap, the authors introduced PARADISE, a large-scale dataset derived from wikiHow. It contains over 104,000 warnings and tips across nine diverse categories, ranging from "Home & Garden" to "Finance & Business." Unlike previous datasets that focus on extracting sequential steps, PARADISE isolates the relationship between a high-level Goal and its associated Warning/Tip. This structure allows for a rigorous test of a model's abductive reasoning capabilities through a multiple-choice Q&A format.

Methodology

The study evaluated two primary model classes:

  1. Fine-tuned Pre-trained Language Models (PLMs): Encoder-based models (e.g., BERT, DeBERTa) trained specifically on the PARADISE training set.
  2. Zero-shot Large Language Models (LLMs): Decoder-based models (e.g., GPT-4, Llama-2) tested using zero-shot prompting to evaluate their inherent procedural knowledge.

To investigate whether models rely on simple word-matching heuristics, the researchers conducted keyword manipulation experiments, where overlapping words between the goal and the correct answer were removed or replaced.

Results and Discussion

The evaluation revealed several key findings regarding the current state of procedural reasoning in AI:

Performance Benchmarks

While humans achieve near-perfect scores (94-96%), AI models still lag behind. Interestingly, fine-tuned smaller models like DeBERTa outperformed massive zero-shot models, reaching up to 93.68% accuracy on tips. Among zero-shot LLMs, GPT-4 led the pack but remained below the performance of specialized smaller architectures.

Figure 4: Accuracy of various models on Warning and Tip inference tasks, comparing original performance with performance after keyword dropping.

Keyword Dependency and Robustness

As shown in Figure 4, all models exhibit a performance drop when common keywords are removed. PLMs saw a significant decline of 15-20%, while LLMs proved more robust but still suffered a 5-15% loss. This suggests that while models possess some procedural awareness, they remain heavily influenced by lexical overlap.

Domain-Specific Failures

Error analysis highlighted a "Physical vs. Abstract" divide. DeBERTa struggled with tangible, physical goals (e.g., tile installation), whereas GPT-4 had more difficulty with abstract social or digital tasks (e.g., workplace dynamics). Additionally, training on "tips" was found to provide better transfer knowledge than training on "warnings," likely because tips are more goal-specific while warnings often involve general safety advice.

Conclusion

The PARADISE benchmark highlights that while LLMs are becoming more capable, they still lack the nuanced implicit planning skills of humans. The superior performance of fine-tuned smaller models suggests that procedural awareness can be effectively specialized. Future work should move beyond multiple-choice selection toward generative reasoning, where models must explain the underlying logic of their procedural advice.

Summary updated

BibTeX
@inproceedings{uzunoglu2024paradise,
  author = {Arda Uzunoglu and Gözde Gül Sahin and Abdulfattah Safa},
  title = {{PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.599},
}
Figure