An Analysis of Procedure Learning by Instruction

Jim BlytheAAAI 2005

Abstract

As automated systems and intelligent assistants become more prevalent in office and computing environments, the ability for non-technical users to customize complex procedures becomes critical. This paper introduces a formal framework to evaluate the effectiveness of "instruction templates"—natural-language-like patterns that map to low-level syntactic transforms. By analyzing these templates through the lenses of coverage, ambiguity, and efficiency, we demonstrate that a compact set of well-designed instructions can provide comprehensive control over task execution systems, significantly reducing the complexity of procedure modification for the end-user.

Introduction

Modern intelligent systems rely on "procedure definitions," which are essentially scripts governing tasks such as invoice processing or equipment procurement. While these procedures are robust, they are often inaccessible to end-users who lack knowledge of the underlying code syntax or the specific technical ontology. Traditional approaches to customization, such as Programming by Demonstration (PBD), often require numerous concrete examples and struggle with abstract logic.

This research addresses the challenge of "Learning by Instruction," where users provide high-level commands (e.g., "Email me if the cost exceeds $400") to modify existing workflows. The primary goal is to establish a measurable standard for determining if a set of instructions is sufficient, clear, and efficient for task modification.

The Tailor Framework

The proposed framework bridges the gap between human intent and machine execution through a three-layered architecture. At the base are Syntactic Transform Operators, a complete set of low-level operations capable of transforming any task expression (e.g., adding a task, deleting a branch). Above this layer are Instruction Templates, which are user-facing patterns like [don’t] do <task> [if <condition>].

The process of mapping user instructions to procedure modifications via template matching and interpretation.

The mapping between these layers is handled by a system called Tailor. As shown in the figure above, Tailor matches user instructions against a template library, interprets the results based on the current procedure state, and applies the corresponding modifications to the procedure definition. This context-dependent interpretation allows the system to resolve ambiguities—such as determining if a term refers to a function argument or a standalone task—by examining the existing workflow.

Evaluation Metrics

To assess the quality of an instruction set, the framework utilizes three primary metrics:

  1. Coverage: The proportion of possible syntactic changes achievable through the available templates. High coverage ensures the user is not "locked out" of necessary modifications.
  2. Ambiguity: The degree of clarification required from the user. The framework distinguishes between term ambiguity (referential uncertainty) and template ambiguity (structural uncertainty regarding the intended operation).
  3. Efficiency: The "compression" provided by templates. High efficiency indicates that a single user instruction performs the work of multiple low-level syntactic transforms, acting essentially as a high-level macro.

Results and Key Insights

The framework was evaluated using the SPARK procedure syntax within an office purchase scenario. Key findings include:

  • Complete Coverage: A surprisingly small set of templates (approximately 3-4 core types) was found to achieve complete coverage of the reference modification set, suggesting that complex language is not a prerequisite for powerful control.
  • Expression Efficiency: The system achieved an efficiency rating of 1.6. This indicates that the average instruction is 60% more powerful than a raw syntactic command, effectively reducing the user's cognitive and manual workload.
  • Ambiguity Isolation: By separating term and template ambiguity, the research shows that template design primarily affects structural clarity. This allows developers to optimize the system's "language" independently of its specific vocabulary.

Conclusion and Future Work

This analysis demonstrates that instruction templates are a viable and efficient method for procedure learning. While the mathematical framework is robust, future work must focus on user realism through empirical studies to ensure templates align with natural human communication. Additionally, improving dialog management will be essential for handling the inherent ambiguities of natural language in more complex, real-world environments.

Summary updated

BibTeX
@inproceedings{blythe2005analysis,
  author = {Jim Blythe},
  title = {{An Analysis of Procedure Learning by Instruction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2005},
}
Figure