Learning from Guidelines: Structured Prompt Optimization for Expert Annotation Tasks

Wenliang Zhong, Haiqing Li, Thao M. Dang, Feng Jiang, Hehuan Ma, Yuzhi Guo, Jean Gao, Junzhou HuangAAAI 2026 · OpenReview

Abstract

Deep learning has significantly advanced numerous fields by training on extensive annotated datasets. However, this data-driven paradigm faces limitations such as limited adaptability and high annotation costs, particularly when precise adherence to detailed, domain-specific guidelines is required. This challenge raises a critical question: Can models effectively shift from data-driven learning to autonomously leveraging guidelines with minimal annotated examples? To address this, we propose the Guideline-Driven Prompt (GDP) optimization framework, which shifts the learning paradigm from data-driven training to guideline-driven reasoning. GDP leverages Retrieval Augmented Generation (RAG) to retrieve essential fragments from complex guidelines and synthesize them into structured, executable prompts. A tree-based optimization algorithm systematically constructs and refines these prompts, explicitly capturing the intricate logic embedded in professional guidelines through a latent pipeline structure. Empirical evaluations on four datasets demonstrate that GDP effectively transitions the learning process from data-intensive methods to a guideline-driven approach, reducing dependence on extensive annotated datasets.

Introduction

In specialized domains like medicine, law, or cybersecurity, data annotation is a complex task governed by extensive, multi-page guidelines. Traditionally, models are trained using a data-driven approach, requiring thousands of human-labeled examples. This process is not only expensive but also slow. While Large Language Models (LLMs) possess strong instruction-following capabilities, they often struggle with guideline-intensive tasks due to context overload from long manuals and a lack of structured reasoning in "flat" prompts.

The Guideline-Driven Prompt (GDP) framework aims to bridge this gap by enabling LLMs to learn in a "human-style": by reading a manual, practicing with a few examples, and refining their understanding.

The GDP Framework

The GDP framework introduces a shift from traditional data-driven learning to a guideline-driven paradigm. As shown in the comparison below, while traditional methods rely on large volumes of annotation data, GDP utilizes a Retrievable Augmented System (RAS) and practice examples to drive LLM predictions, mimicking the way humans learn from guidelines.

Comparison of Human Style, Data-driven, and Guideline-driven learning paradigms.

The architecture is built upon three core pillars:

  1. Guidelines as Knowledge Bases: Instead of overwhelming the LLM with a complete manual, GDP segments guidelines into a Retrievable Augmented System (RAS). The model queries this knowledge base for specific rules relevant to its current task or errors.
  2. Structured Procedural Prompts: GDP moves away from "flat" instructions, instead representing tasks as a pipeline of discrete steps. This structured approach mirrors the multi-stage logical reasoning used by human experts.
  3. Tree-Guided Optimization: To refine these prompts, GDP employs a tree-based search algorithm. This algorithm iteratively improves the prompt by either decomposing complex instructions into sub-tasks or optimizing the content of existing steps based on feedback.
The tree-based optimization algorithm used to systematically construct and refine prompts.

Experimental Evaluation

The authors evaluated GDP on four guideline-intensive information extraction datasets: NCBI-Disease (Medical NER), WNUT (News entities), HarveyNER (Disaster-related tweets), and CASIE (Cybercrime events).

Key Results

  • Superior Few-Shot Performance: In 5-shot settings, GDP achieved an average F1 score of 62.9% on GPT-4, significantly outperforming standard prompting and other optimization methods like PromptAgent.
  • Structural Importance: A key finding was that the structure of the prompt (the pipeline) often matters more than specific wording. Breaking tasks into sequential steps significantly reduces model confusion.
  • Efficiency: GDP demonstrates that a well-structured prompt derived from a guideline can match or exceed the performance of models trained on much larger labeled datasets without guideline access.

Conclusion

The GDP framework represents a significant step toward autonomous, expert-level annotation. By shifting the focus from data-heavy fine-tuning to structured, guideline-driven reasoning, GDP reduces the reliance on expensive labeled datasets while maintaining high accuracy in complex, domain-specific tasks. Future work will explore multi-modal guidelines and dynamic updates to prompts as professional regulations evolve.

Summary updated

Abstract

Deep learning has significantly advanced numerous fields by training on extensive annotated datasets. However, this data-driven paradigm faces limitations such as limited adaptability and high annotation costs, particularly when precise adherence to detailed, domain-specific guidelines is required in annotation. This challenge raises a critical question: Can models effectively shift from data-driven learning to autonomously leveraging guidelines with minimal annotated examples? To address this, we propose the Guideline-Driven Prompt (GDP) optimization framework, which shifts the learning paradigm from data-driven training to guideline-driven reasoning. GDP leverages Retrieval Augmented Generation (RAG) to retrieve essential fragments from complex guidelines and synthesize them into structured, executable prompts. A tree-based optimization algorithm systematically constructs and refines these prompts, explicitly capturing the intricate logic embedded in professional guidelines through a latent pipeline structure. Empirical evaluations on four datasets ranging from diverse domains and different tasks demonstrate that GDP effectively transitions the learning process from data-intensive methods to a guideline-driven approach in tasks requiring detailed and complex guideline adherence, reducing dependence on extensive annotated datasets.
BibTeX
@inproceedings{zhong2026learning,
  author = {Wenliang Zhong and Haiqing Li and Thao M. Dang and Feng Jiang and Hehuan Ma and Yuzhi Guo and Jean Gao and Junzhou Huang},
  title = {{Learning from Guidelines: Structured Prompt Optimization for Expert Annotation Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=OuTXzE3mDS},
}
Figure