Abstract
How-to procedures are essential for millions of users, yet generic instructions often fail to meet specific individual needs, such as adapting a gardening guide for an apartment setting. This paper investigates and improves Large Language Model (LLM) performance in procedure customization. We introduce CustomPlans, a new evaluation benchmark of over 200 WikiHow procedures paired with diverse customization hints. Our research demonstrates that a multi-agent architecture using sequential LLM agents—one to edit the procedure and one to verify its executability—significantly outperforms standard end-to-end prompting by 10.5% in absolute correctness. This suggests that structured, edit-based multi-agent frameworks are a robust path forward for tailoring complex procedural knowledge.
The Challenge of Procedural Customization
From "how to plant a garden" to "how to bake bread," procedural knowledge is a cornerstone of web search. However, generic instructions rarely fit every user's specific context. A user might live in a small apartment, have a gluten allergy, or lack specific tools.
While LLMs are adept at generating fluent text, they often struggle to customize existing procedures accurately. When tasked with adapting a plan, standard LLMs frequently produce "uncustomized" or "inexecutable" outputs. They may add unnecessary steps, omit crucial ones, or hallucinate entirely new structures that ignore the original logic. Because LLMs generate text autoregressively, they often lack the precision required to "edit" a plan the way a human would, leading to significant model drift.
Methods: Disentangling Customization via Semi-Symbolic Edits
To address these failures, we propose moving away from "End-to-End" (E2E) prompting. Instead, we disentangle the task into two distinct roles handled by a multi-agent framework:
- The Modify Agent: Focuses solely on suggesting edits to address the user's specific customization hint (e.g., "I don't have an oven").
- The Verify Agent: Reviews the modified plan to ensure it remains executable and adds any missing details required by the changes.
Crucially, these agents communicate using semi-symbolic edits. Instead of rewriting the whole procedure, they output structured commands like insert(step_number, text) to add new steps or replace(step_number, text) to update or delete existing ones.
We explored three primary interaction patterns for these agents:
- Unified: A single agent suggests all edits at once.
- Sequential: The Modify agent suggests changes, then the Verify agent cleans them up.
- Parallel: Both agents suggest edits simultaneously, and a third "Resolution" module merges them.
The CUSTOMPLANS Benchmark
We introduced CUSTOMPLANS, a benchmark consisting of 206 goals across 9 diverse domains. It includes CUSTOMPLANS-Real, featuring human-authored customization hints, and CUSTOMPLANS-Simulated, derived from existing datasets where constraints are treated as customization hints. The dataset categorizes needs by type (Prerequisite, Preference, or Refinement) and user expertise level (Beginner to Expert).
Key Insights and Results
Our evaluation, conducted by human experts, focused on whether the resulting procedures were both Customized (met the user's hint) and Executable (logically sound and performable).
- Sequential Agents are Superior: The Sequential architecture was the clear winner, producing 51.94% "Fully Correct" procedures. In contrast, standard End-to-End (E2E) prompting achieved only 41.46%.
- Executability Boost: The agentic framework was significantly better at maintaining realism. The Sequential model reached 72.33% executability, compared to a dismal 48.45% for the standard E2E approach.
- Editing > Rewriting: By forcing the model to use structured operations, the system preserves the inherent structure of the original valid procedure, reducing hallucinations.
- Prerequisites are the Hardest: Models struggle most with "hard" constraints (e.g., "I only have a stovetop") compared to "soft" preferences. Hard constraints often require complex logical changes across the entire procedure.
Conclusion
Our findings suggest that LLMs can be configured effectively for procedure customization by using specialized, sequential agents. The success of the "Modify then Verify" approach highlights the importance of agent ordering and structured editing. Future work may explore how these multi-agent architectures can be scaled to other customization applications, such as coding or creative writing, and how to automate the discovery of optimal agent sequences.
Summary updated