GPS: Genetic Prompt Search for Efficient Few-shot Learning

Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang

Abstract

Finding the optimal natural language prompt for Large Language Models (LLMs) is a critical but labor-intensive process. This paper presents Genetic Prompt Search (GPS), a gradient-free, tuning-free method that automates prompt discovery using an evolutionary algorithm. By treating prompt engineering as a discrete search problem, GPS utilizes generative models to mutate candidate prompts and selects the most effective versions based on a minimal validation set. Our results demonstrate that GPS consistently outperforms manual prompt engineering and rivals parameter-intensive tuning methods while maintaining the efficiency of a frozen model architecture.

Introduction

Large Language Models (LLMs) exhibit remarkable few-shot capabilities, yet their performance is highly sensitive to the specific phrasing of a prompt. This sensitivity makes "prompt engineering" a bottleneck in deploying LLMs for specialized tasks. Currently, practitioners rely on manual trial-and-error or soft prompt tuning. The former is time-consuming and often misses optimal configurations, while the latter requires access to model gradients and significant storage for learned embeddings—requirements that are often incompatible with proprietary models accessed via APIs.

GPS (Genetic Prompt Search) addresses these challenges by searching the space of natural language prompts directly. By keeping the underlying model frozen and avoiding gradient-based updates, GPS provides a lightweight and interpretable alternative to traditional fine-tuning.

Methodology

GPS treats the discovery of high-performing prompts as an evolutionary optimization problem. The algorithm operates through a classic genetic loop across several generations:

1. Initialization

The process begins with a small set of handcrafted "seed" prompts designed for the target task.

2. Prompt Generation (Mutation)

To explore the linguistic space, GPS generates variations of existing prompts using three primary mutation strategies:

  • Back Translation (BT): Prompts are translated into a secondary language (e.g., Chinese or Spanish) and then back to English to introduce semantic and syntactic variations.
  • Cloze: A T5 model is used to fill in blanks within a prompt template, generating diverse phrasing.
  • Sentence Continuation (SC): A generative model (such as T5LM-XXL) is tasked with rephrasing the prompt based on a prefix like "Write two sentences that mean the same thing."

3. Scoring and Selection

Each candidate prompt is evaluated against a "true few-shot" validation set, typically consisting of only 32 labeled examples. The top $K$ performing prompts are selected to serve as the parents for the next generation.

4. Iteration

This cycle of mutation and selection repeats for a fixed number of generations (usually six), progressively refining the prompts toward higher accuracy.

Experimental Setup

We evaluated GPS using the T0 benchmark, focusing on 10 diverse NLP tasks that the model had not encountered during its multitask pretraining. These tasks spanned several categories:

  • Natural Language Inference (NLI): ANLI (R1, R2, R3), CB, and RTE.
  • Coreference Resolution: WSC and Winogrande.
  • Sentence Completion/Commonsense: COPA and HellaSwag.
  • Word Sense Disambiguation: WiC.

The search process was conducted using only 32 samples per task, ensuring the method remains viable in data-scarce environments.

Key Results and Discussion

Performance Gains

GPS demonstrated a significant improvement over baseline methods. On average, the prompts discovered by GPS outperformed manual prompts by 2.6 percentage points across the 10 datasets. With an average accuracy of 60.12%, GPS surpassed other discrete search methods like GRIPS (58.66%) and approached the performance of full Model Tuning (61.73%).

Efficiency and Interpretability

One of the primary advantages of GPS is its computational efficiency. In terms of equivalent forward passes, GPS is approximately 11 times more efficient than standard Model Tuning or Soft Prompt Tuning. Furthermore, unlike methods that produce unintelligible "trigger words," GPS results in human-readable, semantically fluent prompts. For example, in the WSC task, GPS improved accuracy by 10 points simply by removing the phrase "In the passage above" and adding the clarifying hint "the person of."

Mutation Strategy Efficacy

Among the mutation strategies, Sentence Continuation using T5LM-XXL proved most effective. This strategy produced prompts that were both semantically diverse and linguistically coherent, providing the best candidates for the evolutionary search.

Limitations and Future Work

While GPS shows great promise, it is currently limited by its reliance on a small validation set, which may lead to overfitting if the 32 samples are not representative of the broader task. Additionally, while the method identifies what works, it does not yet explain why specific linguistic nuances trigger better performance in LLMs. Future research will focus on scaling GPS to more complex reasoning tasks and exploring the underlying mechanisms of prompt sensitivity to further refine the search space.

Conclusion

GPS offers a robust, tuning-free framework for optimizing LLM performance in few-shot scenarios. By automating the prompt engineering process through genetic search, it achieves high accuracy with minimal computational overhead, making it an ideal solution for optimizing models behind restricted APIs.

Summary updated

BibTeX
@inproceedings{xu2022gps,
  author = {Hanwei Xu and Yujun Chen and Yulun Du and Nan Shao and Yanggang Wang and Haiyu Li and Zhilin Yang},
  title = {{GPS: Genetic Prompt Search for Efficient Few-shot Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=oGHUoZFAbB9},
}
Figure