Abstract
Deep learning has significantly advanced numerous fields by training on extensive annotated datasets. However, this data-driven paradigm faces limitations such as limited adaptability and high annotation costs, particularly when precise adherence to detailed, domain-specific guidelines is required. This challenge raises a critical question: Can models effectively shift from data-driven learning to autonomously leveraging guidelines with minimal annotated examples? To address this, we propose the Guideline-Driven Prompt (GDP) optimization framework, which shifts the learning paradigm from data-driven training to guideline-driven reasoning. GDP leverages Retrieval Augmented Generation (RAG) to retrieve essential fragments from complex guidelines and synthesize them into structured, executable prompts. A tree-based optimization algorithm systematically constructs and refines these prompts, explicitly capturing the intricate logic embedded in professional guidelines through a latent pipeline structure. Empirical evaluations on four datasets demonstrate that GDP effectively transitions the learning process from data-intensive methods to a guideline-driven approach, reducing dependence on extensive annotated datasets.
Introduction
In specialized domains like medicine, law, or cybersecurity, data annotation is a complex task governed by extensive, multi-page guidelines. Traditionally, models are trained using a data-driven approach, requiring thousands of human-labeled examples. This process is not only expensive but also slow. While Large Language Models (LLMs) possess strong instruction-following capabilities, they often struggle with guideline-intensive tasks due to context overload from long manuals and a lack of structured reasoning in "flat" prompts.
The Guideline-Driven Prompt (GDP) framework aims to bridge this gap by enabling LLMs to learn in a "human-style": by reading a manual, practicing with a few examples, and refining their understanding.
The GDP Framework
The GDP framework introduces a shift from traditional data-driven learning to a guideline-driven paradigm. As shown in the comparison below, while traditional methods rely on large volumes of annotation data, GDP utilizes a Retrievable Augmented System (RAS) and practice examples to drive LLM predictions, mimicking the way humans learn from guidelines.
The architecture is built upon three core pillars:
- Guidelines as Knowledge Bases: Instead of overwhelming the LLM with a complete manual, GDP segments guidelines into a Retrievable Augmented System (RAS). The model queries this knowledge base for specific rules relevant to its current task or errors.
- Structured Procedural Prompts: GDP moves away from "flat" instructions, instead representing tasks as a pipeline of discrete steps. This structured approach mirrors the multi-stage logical reasoning used by human experts.
- Tree-Guided Optimization: To refine these prompts, GDP employs a tree-based search algorithm. This algorithm iteratively improves the prompt by either decomposing complex instructions into sub-tasks or optimizing the content of existing steps based on feedback.
Experimental Evaluation
The authors evaluated GDP on four guideline-intensive information extraction datasets: NCBI-Disease (Medical NER), WNUT (News entities), HarveyNER (Disaster-related tweets), and CASIE (Cybercrime events).
Key Results
- Superior Few-Shot Performance: In 5-shot settings, GDP achieved an average F1 score of 62.9% on GPT-4, significantly outperforming standard prompting and other optimization methods like PromptAgent.
- Structural Importance: A key finding was that the structure of the prompt (the pipeline) often matters more than specific wording. Breaking tasks into sequential steps significantly reduces model confusion.
- Efficiency: GDP demonstrates that a well-structured prompt derived from a guideline can match or exceed the performance of models trained on much larger labeled datasets without guideline access.
Conclusion
The GDP framework represents a significant step toward autonomous, expert-level annotation. By shifting the focus from data-heavy fine-tuning to structured, guideline-driven reasoning, GDP reduces the reliance on expensive labeled datasets while maintaining high accuracy in complex, domain-specific tasks. Future work will explore multi-modal guidelines and dynamic updates to prompts as professional regulations evolve.
Summary updated