Introduction
Large Language Models (LLMs) are increasingly deployed as autonomous planners. However, standard evaluations often rely on simplified environments like BlocksWorld, which fail to capture the linguistic complexity of real-world tasks. A critical component of effective planning is "implicit knowledge"—the ability to anticipate potential pitfalls (warnings) or best practices (tips) associated with a goal, even without an explicit step-by-step instruction. For instance, when tasked with "Removing Ink Stains from Fabric," a proficient planner should inherently know to blot rather than rub the stain.
The PARADISE Dataset
To address this gap, the authors introduced PARADISE, a large-scale dataset derived from wikiHow. It contains over 104,000 warnings and tips across nine diverse categories, ranging from "Home & Garden" to "Finance & Business." Unlike previous datasets that focus on extracting sequential steps, PARADISE isolates the relationship between a high-level Goal and its associated Warning/Tip. This structure allows for a rigorous test of a model's abductive reasoning capabilities through a multiple-choice Q&A format.
Methodology
The study evaluated two primary model classes:
- Fine-tuned Pre-trained Language Models (PLMs): Encoder-based models (e.g., BERT, DeBERTa) trained specifically on the PARADISE training set.
- Zero-shot Large Language Models (LLMs): Decoder-based models (e.g., GPT-4, Llama-2) tested using zero-shot prompting to evaluate their inherent procedural knowledge.
To investigate whether models rely on simple word-matching heuristics, the researchers conducted keyword manipulation experiments, where overlapping words between the goal and the correct answer were removed or replaced.
Results and Discussion
The evaluation revealed several key findings regarding the current state of procedural reasoning in AI:
Performance Benchmarks
While humans achieve near-perfect scores (94-96%), AI models still lag behind. Interestingly, fine-tuned smaller models like DeBERTa outperformed massive zero-shot models, reaching up to 93.68% accuracy on tips. Among zero-shot LLMs, GPT-4 led the pack but remained below the performance of specialized smaller architectures.
Keyword Dependency and Robustness
As shown in Figure 4, all models exhibit a performance drop when common keywords are removed. PLMs saw a significant decline of 15-20%, while LLMs proved more robust but still suffered a 5-15% loss. This suggests that while models possess some procedural awareness, they remain heavily influenced by lexical overlap.
Domain-Specific Failures
Error analysis highlighted a "Physical vs. Abstract" divide. DeBERTa struggled with tangible, physical goals (e.g., tile installation), whereas GPT-4 had more difficulty with abstract social or digital tasks (e.g., workplace dynamics). Additionally, training on "tips" was found to provide better transfer knowledge than training on "warnings," likely because tips are more goal-specific while warnings often involve general safety advice.
Conclusion
The PARADISE benchmark highlights that while LLMs are becoming more capable, they still lack the nuanced implicit planning skills of humans. The superior performance of fine-tuned smaller models suggests that procedural awareness can be effectively specialized. Future work should move beyond multiple-choice selection toward generative reasoning, where models must explain the underlying logic of their procedural advice.
Summary updated