Abstract
How-to procedures, such as how to plant a garden, are now used by millions of users, but sometimes need customizing to meet a user's specific needs, e.g., planting a garden without pesticides. Our goal is to measure and improve an LLM's ability to perform such customization. Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need. We find that a simple architecture with two LLM agents used sequentially performs best, one that edits a generic how-to procedure and one that verifies its executability, significantly outperforming (10.5% absolute) an end-to-end prompted LLM. This suggests that LLMs can be configured reasonably effectively for procedure customization. This also suggests that multi-agent editing architectures may be worth exploring further for other customization applications (e.g. coding, creative writing) in the future.
Introduction
Large Language Models (LLMs) have become ubiquitous tools for generating "how-to" guides. However, these models often struggle when faced with specific user constraints—a scenario where "one size does not fit all." For instance, a general guide on "how to plant a garden" becomes problematic if the user specifies, "I live in an apartment." Standard end-to-end (E2E) prompting frequently results in "structural drift," where the model rewrites the entire procedure and loses the original logic, or "execution errors," where the model satisfies the constraint but breaks the fundamental steps of the task.
This research addresses these challenges by moving away from free-form re-generation and toward a framework of targeted, semi-symbolic edits managed by multi-agent architectures.
The CustomPlans Dataset
To evaluate customization capabilities, the researchers introduced CustomPlans, a benchmark consisting of 206 WikiHow-based procedures. This dataset combines human-collected data with machine-generated, human-verified scripts across nine diverse domains.
As shown in the figure above, the dataset categorizes user hints into three primary dimensions:
- Expertise: Ranging from beginner (66.8%) to expert (10.4%).
- Constraint: Specific limitations such as "I have an allergy" or "I don't have a stove."
- Critical Type: Identifying which aspect of the plan is most vital to modify.
Methodology: Semi-Symbolic Edits
The core innovation of this work is the use of semi-symbolic edits rather than full-text generation. Instead of asking an LLM to write a new guide from scratch, the system is prompted to suggest specific operations:
insert(step_number, text): Adding a necessary new step.replace(step_number, text): Modifying or deleting an existing step to fit the constraint.
This structured format acts as a regularizer, forcing the model to stay grounded in the original plan's structure. This significantly reduces hallucinations and the introduction of unnecessary "extra steps" that often plague end-to-end models.
Multi-Agent Architectures
The study explored three distinct interaction patterns for these edit-based agents:
- Unified: A single agent suggests all edits for both customization and executability simultaneously.
- Sequential: A
Modifyagent first suggests edits to meet the user's hint. Subsequently, aVerifyagent reviews the modified plan and suggests further edits to ensure the procedure remains executable. This "customize first, fix later" approach avoids cognitive overhead. - Parallel: Both agents suggest edits independently, which are then merged by an "Edit Resolution" module to resolve potential conflicts.
Results and Key Insights
The Sequential architecture emerged as the superior method, achieving a 51.94% full correctness rate—a 10.5% absolute improvement over standard E2E prompting (41.46%).
Key findings include:
- Executability: Edit-based agents produced executable plans over 70% of the time, while E2E models fell below 50%.
- Constraint Difficulty: "Prerequisite" constraints (e.g., lacking a specific tool) proved much harder for models to satisfy than simple "Preferences," as they require more fundamental changes to the procedure's underlying logic.
- Precision over Creativity: While E2E models are more creative, this trait is a disadvantage in procedural tasks where maintaining proven, safe steps is paramount.
Conclusion
By separating the act of customization from the act of verification, multi-agent sequential architectures can significantly improve the reliability of LLM-generated procedures. This edit-based approach provides a robust framework for tailoring complex information to specific user needs without sacrificing the integrity of the original instructions. Future work may extend this methodology to other structured domains such as code customization and collaborative writing.
Summary updated