APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching

Kun Qian, Yisi Sang, Farima Fatahi Bayat, Anton Belyi, Xianqi Chu, Yash Govind, Samira Khorshidi, Rahul Khot, Katherine Luna, Azadeh Nikfarjam, Xiaoguang Qi, Fei Wu, Xianhan Zhang, Yunyao Li

TL;DR

Prompt engineering for Large Language Models (LLMs) is crucial but time-consuming, especially when selecting few-shot examples. This paper introduces APE, an active learning-based tool that interactively helps users find the most informative examples to improve LLM performance on specific tasks, making prompt engineering more efficient.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful, but getting them to perform specific tasks reliably often requires "prompt engineering." This involves crafting instructions and providing a few examples (few-shot learning) to guide the LLM. While this is generally easier than fine-tuning an LLM, the process of selecting the most effective few-shot examples is a significant bottleneck. Imagine having 100 potential examples and needing to pick just three – the number of combinations is astronomical, making manual selection a tedious and inefficient task. This paper addresses the challenge of efficiently identifying these crucial few-shot examples to boost LLM performance.

Methods

APE (Active Prompt Engineering) is a human-in-the-loop tool that leverages active learning principles to streamline the selection of informative few-shot examples. The core idea is to iteratively present the LLM with a prompt and a pool of potential examples.

Here's how it works:

  • Active Sampling: APE employs sampling strategies to identify examples that the LLM is most "uncertain" about. This uncertainty is a proxy for informativeness. The paper discusses two task-agnostic strategies:
  • Random-based Sampling: Simply picks a random subset of examples from the pool. This is fast but less effective for complex tasks.
  • Self-consistency-based Sampling: This more sophisticated method is inspired by "query-by-committee" and "self-consistency" techniques. For a given example, the LLM processes the prompt multiple times, potentially with slightly varied "temperatures" (controlling creativity vs. determinism). The disagreement among these multiple outputs is used to gauge uncertainty. Examples with high disagreement are considered more informative.
  • Human Annotation: The selected uncertain examples are presented to a human user for annotation. This can include providing the correct label and, optionally, explanations (e.g., for Chain-of-Thought prompting).
  • Prompt Updating: The newly annotated examples are then incorporated into the LLM's prompt for the next iteration.
  • Evaluation: The updated prompt's performance is evaluated.

The system is designed with a user-friendly interface to abstract away the underlying active learning complexities.

Datasets

The paper mentions using the DBLP-Scholar dataset sampled from (Köpcke et al., 2010) for their entity matching demo. This dataset is used for training and/or evaluation.

Key Insights and Ideas

  • Active Learning for Prompt Engineering: The paper frames few-shot example selection as an active learning problem, which is a novel and promising direction for improving LLM efficiency.
  • Uncertainty as a Signal: The core insight is that examples where the LLM is most uncertain are likely the most informative for improving its performance. This allows for targeted human effort.
  • User-Centric Tooling: APE emphasizes building a usable tool with an intuitive interface, recognizing that practical adoption requires more than just a good algorithm.
  • Self-Consistency for Uncertainty: Using multiple LLM runs with varying temperatures to measure disagreement is an effective way to identify ambiguous examples without needing multiple distinct LLM models.

Key Results

The paper focuses on the tooling aspect and the methodology. While specific quantitative results on LLM performance improvement are not detailed in this demo paper, the authors demonstrate a functional tool that iteratively refines prompts by incorporating human-annotated, informative few-shot examples. The demo video (linked in the abstract) would likely showcase the interactive process and the types of examples APE selects.

Limitations and Future Work

  • Focus on Tooling: The authors explicitly state that this demo paper focuses on the tooling aspect and that a future research paper will provide a comprehensive description of the system design, theoretical foundation, and experimental evaluations.
  • Task-Agnostic Strategies: While task-agnostic strategies are presented for broader applicability, the paper acknowledges that task-specific strategies could also be integrated.
  • Human Annotation Effort: While APE aims to reduce manual effort, human annotation is still a necessary component. The efficiency of this annotation process could be a point for future optimization.
  • Experimental Validation: The paper lacks detailed experimental results to quantify the effectiveness of APE compared to random selection or other methods. This is a clear area for future work.

Summary updated

Abstract

Prompt engineering is an iterative procedure often requiring extensive manual effort to formulate suitable instructions for effectively directing large language models (LLMs) in specific tasks. Incorporating few-shot examples is a vital and effective approach to providing LLMs with precise instructions, leading to improved LLM performance. Nonetheless, identifying the most informative demonstrations for LLMs is labor-intensive, frequently entailing sifting through an extensive search space. In this demonstration, we showcase a human-in-the-loop tool called APE (Active Prompt Engineering) designed for refining prompts through active learning. Drawing inspiration from active learning, APE iteratively selects the most ambiguous examples for human feedback, which will be transformed into few-shot examples within the prompt. The demo recording can be found with the submission or be viewed at https://youtu.be/OwQ6MQx53-Y.
BibTeX
@inproceedings{qian2024ape,
  author = {Kun Qian and Yisi Sang and Farima Fatahi Bayat and Anton Belyi and Xianqi Chu and Yash Govind and Samira Khorshidi and Rahul Khot and Katherine Luna and Azadeh Nikfarjam and Xiaoguang Qi and Fei Wu and Xianhan Zhang and Yunyao Li},
  title = {{APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure