Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization

Yash Kumar Lal, Li Zhang, Faeze Brahman, Bodhisattwa Prasad Majumder, Peter Clark, Niket Tandon

Abstract

How-to procedures are essential for millions of users, yet generic instructions often fail to meet specific individual needs, such as adapting a gardening guide for an apartment setting. This paper investigates and improves Large Language Model (LLM) performance in procedure customization. We introduce CustomPlans, a new evaluation benchmark of over 200 WikiHow procedures paired with diverse customization hints. Our research demonstrates that a multi-agent architecture using sequential LLM agents—one to edit the procedure and one to verify its executability—significantly outperforms standard end-to-end prompting by 10.5% in absolute correctness. This suggests that structured, edit-based multi-agent frameworks are a robust path forward for tailoring complex procedural knowledge.

The Challenge of Procedural Customization

From "how to plant a garden" to "how to bake bread," procedural knowledge is a cornerstone of web search. However, generic instructions rarely fit every user's specific context. A user might live in a small apartment, have a gluten allergy, or lack specific tools.

While LLMs are adept at generating fluent text, they often struggle to customize existing procedures accurately. When tasked with adapting a plan, standard LLMs frequently produce "uncustomized" or "inexecutable" outputs. They may add unnecessary steps, omit crucial ones, or hallucinate entirely new structures that ignore the original logic. Because LLMs generate text autoregressively, they often lack the precision required to "edit" a plan the way a human would, leading to significant model drift.

Methods: Disentangling Customization via Semi-Symbolic Edits

To address these failures, we propose moving away from "End-to-End" (E2E) prompting. Instead, we disentangle the task into two distinct roles handled by a multi-agent framework:

  1. The Modify Agent: Focuses solely on suggesting edits to address the user's specific customization hint (e.g., "I don't have an oven").
  2. The Verify Agent: Reviews the modified plan to ensure it remains executable and adds any missing details required by the changes.

Crucially, these agents communicate using semi-symbolic edits. Instead of rewriting the whole procedure, they output structured commands like insert(step_number, text) to add new steps or replace(step_number, text) to update or delete existing ones.

Comparison between end-to-end prompting and the proposed semi-symbolic edit approach.

We explored three primary interaction patterns for these agents:

  • Unified: A single agent suggests all edits at once.
  • Sequential: The Modify agent suggests changes, then the Verify agent cleans them up.
  • Parallel: Both agents suggest edits simultaneously, and a third "Resolution" module merges them.

The CUSTOMPLANS Benchmark

We introduced CUSTOMPLANS, a benchmark consisting of 206 goals across 9 diverse domains. It includes CUSTOMPLANS-Real, featuring human-authored customization hints, and CUSTOMPLANS-Simulated, derived from existing datasets where constraints are treated as customization hints. The dataset categorizes needs by type (Prerequisite, Preference, or Refinement) and user expertise level (Beginner to Expert).

Key Insights and Results

Our evaluation, conducted by human experts, focused on whether the resulting procedures were both Customized (met the user's hint) and Executable (logically sound and performable).

The Sequential multi-agent architecture, which proved most effective for procedure customization.
  • Sequential Agents are Superior: The Sequential architecture was the clear winner, producing 51.94% "Fully Correct" procedures. In contrast, standard End-to-End (E2E) prompting achieved only 41.46%.
  • Executability Boost: The agentic framework was significantly better at maintaining realism. The Sequential model reached 72.33% executability, compared to a dismal 48.45% for the standard E2E approach.
  • Editing > Rewriting: By forcing the model to use structured operations, the system preserves the inherent structure of the original valid procedure, reducing hallucinations.
  • Prerequisites are the Hardest: Models struggle most with "hard" constraints (e.g., "I only have a stovetop") compared to "soft" preferences. Hard constraints often require complex logical changes across the entire procedure.

Conclusion

Our findings suggest that LLMs can be configured effectively for procedure customization by using specialized, sequential agents. The success of the "Modify then Verify" approach highlights the importance of agent ordering and structured editing. Future work may explore how these multi-agent architectures can be scaled to other customization applications, such as coding or creative writing, and how to automate the discovery of optimal agent sequences.

Summary updated

Abstract

How-to procedures, such as how to plant a garden, are now used by millions of users, but sometimes need customizing to meet a user’s specific needs, e.g., planting a garden without pesticides. Our goal is to measure and improve an LLM’s ability to perform such customization. Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need. We find that a simple architecture with two LLM agents used sequentially performs best, one that edits a generic how-to procedure and one that verifies its executability, significantly outperforming (10.5% absolute) an end-to-end prompted LLM. This suggests that LLMs can be configured reasonably effectively for procedure customization. This also suggests that multi-agent editing architectures may be worth exploring further for other customization applications (e.g. coding, creative writing) in the future.
BibTeX
@inproceedings{lal2024tailoring,
  author = {Yash Kumar Lal and Li Zhang and Faeze Brahman and Bodhisattwa Prasad Majumder and Peter Clark and Niket Tandon},
  title = {{Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=5XP6upaU0c},
}
Figure