Reordering Examples Helps during Priming-based Few-Shot Learning

Sawan Kumar, Partha TalukdarACL 2021

TL;DR

This paper introduces PERO, a novel method for few-shot learning that treats example ordering as a search problem. By using a genetic algorithm to find the optimal sequence of training examples for a prompt, PERO significantly improves generalization in low-data scenarios, even outperforming models trained on much larger datasets.

What Problem Does This Paper Address?

Few-shot learning, the ability to learn from a very small number of examples, is a crucial capability for making AI systems more adaptable and personalized. While large pre-trained language models (LLMs) have shown promise in few-shot settings by using task descriptions and examples as prompts (priming), their performance can be highly sensitive to how these examples are presented. Existing methods often treat example order as secondary, or rely on complex prompt engineering. This paper addresses the gap of systematically optimizing the order of examples in few-shot prompts to maximize performance without requiring model parameter updates.

Methods

The core of PERO is a genetic algorithm that searches for the optimal permutation of training examples to use as a prompt. Here's how it works:

  • Individuals: Each "individual" in the genetic algorithm represents a specific ordering (permutation) of a small set of training examples.
  • Fitness Function: The fitness of an individual is determined by how well the LLM performs on a validation set when prompted with that specific example order. The metric used is typically cross-entropy loss.
  • Selection, Crossover, and Mutation: Standard genetic algorithm operations are employed. "Fit" individuals (those leading to better performance) are selected to "breed" new permutations. Crossover combines parts of parent permutations, and mutation randomly alters an individual's order.
  • Separator Token Learning (Optional): PERO also explores learning a special "separator token" to concatenate examples within the prompt, rather than relying on a default like a newline. This learned token can further enhance performance.

Crucially, PERO does not fine-tune the LLM's parameters. This makes it a valuable tool for probing the inherent knowledge within pre-trained models.

Datasets

The paper evaluates PERO on three standard NLP tasks:

  • Sentiment Classification: SST-2 dataset.
  • Natural Language Inference (NLI): A dataset derived from SICK-E and SemEval 2012 STS.
  • Fact Retrieval: Datasets from Shin et al. (2020) and LAMA (Petroni et al., 2019).

The experiments primarily focus on using very small training sets (e.g., 10 examples) to demonstrate few-shot capabilities, but also report results with larger datasets for comparison.

Key Insights and Ideas

  • Order Matters (A Lot): The most significant insight is that the sequence in which examples are presented in a few-shot prompt has a profound impact on generalization performance. PERO's success hinges on this observation.
  • Permutation Search is Effective: A genetic algorithm, a biologically inspired search heuristic, is surprisingly effective at finding good example orderings, leading to substantial performance gains.
  • Learned Separator Tokens Can Help: While a newline is a natural separator, learning a task-specific separator token can provide an additional boost in performance.
  • Interpretable Probing: By not fine-tuning the model, PERO offers a way to understand what knowledge is already encoded in LLMs, and how it can be best elicited through carefully crafted prompts.
  • One-Shot Learning Potential: The analysis suggests that with well-chosen examples and learned label patterns, one-shot learning (using only one example per class) can achieve competitive results.

Key Results

  • Significant Gains in Few-Shot Learning: PERO consistently outperforms baselines like Autoprompt (Shin et al., 2020) when using only 10 training examples, often achieving accuracies comparable to or exceeding models trained on thousands of examples. For instance, on sentiment classification with 10 examples, PERO reaches 91.2% accuracy, while Autoprompt achieves 54.73%.
  • Competitive with Larger Datasets: Even when more data is available, PERO remains competitive with other methods, though traditional fine-tuning may surpass it with very large datasets.
  • Fact Retrieval Improvement: PERO shows significant gains in fact retrieval, outperforming Autoprompt on most relations, especially when using limited data.
  • Ablation Studies Confirm Components: Both the genetic algorithm for permutation search and the learned separator token contribute positively to performance, with the permutation search being the primary driver.

Limitations and Future Work

  • Computational Cost: The genetic algorithm, while effective, can be computationally intensive, especially as the number of training examples increases, leading to a larger search space (e.g., 10! permutations for 10 examples).
  • Scalability of Search: The paper acknowledges that for very large numbers of training examples, the search space becomes enormous, and a better search strategy might be needed to improve PERO's performance in such scenarios. This is identified as an area for future work.
  • Interpreting Learned Prompts: While PERO is more interpretable than methods using learned trigger tokens, further analysis of the learned prompts could yield deeper insights.
  • One-Shot Learning Recipes: The paper hints at effective recipes for one-shot learning based on learned label patterns, suggesting this as a promising avenue for future research.

Summary updated

BibTeX
@inproceedings{kumar2021reordering,
  author = {Sawan Kumar and Partha Talukdar},
  title = {{Reordering Examples Helps during Priming-based Few-Shot Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure