Abstract
In-Context Learning (ICL) has traditionally relied on the careful selection of few-shot examples to maximize Large Language Model (LLM) performance. However, the rise of instruction-tuned models raises a critical question: does the quality of the instruction render example optimization obsolete? This paper introduces NICE (Normalized Invariability to Choice of Examples), a metric designed to quantify a model's sensitivity to example selection. Our findings reveal a "flattening effect": as instructions become more detailed, the performance variance across different example sets vanishes for classification tasks. Conversely, complex generative tasks maintain a high sensitivity to example choice, suggesting that the necessity of example optimization is task-dependent.
Introduction
In the evolving landscape of In-Context Learning, a significant amount of research has been dedicated to "example selection"—the process of retrieving the most relevant demonstrations from a training set to include in a prompt. While effective, these retrieval algorithms add significant computational overhead. This paper investigates whether this effort is still justified for modern, instruction-tuned LLMs. By systematically varying instruction detail and measuring performance stability via the NICE metric, we provide a framework for deciding when to optimize examples and when to focus solely on prompt engineering.
The NICE Metric
To measure how much example choice actually matters, the authors propose the Normalized Invariability to Choice of Examples (NICE). The metric is calculated by partitioning candidate examples into "bins" based on their cosine similarity to the query and then comparing the average performance across all bins to the peak performance of the best-performing bin:
[ NICE = \frac{\text{Average Performance}}{\text{Maximum Performance}} ]
- High NICE (approaching 1.0): The model is robust; performance is consistent regardless of which examples are chosen.
- Low NICE (approaching 0.0): The model is highly sensitive; picking the "wrong" examples leads to significant performance degradation.
Experimental Methodology
1. Instruction Hierarchy
The researchers tested models across a spectrum of instruction quality, ranging from no instruction to highly specific rules:
- NI (No Instruction): Only few-shot examples are provided.
- TD (Task Definition): A concise one-sentence description of the task.
- +LS (Label Space): Inclusion of valid output labels (e.g., "Options: Positive, Negative").
- +LM (Label Meanings): Definitions of what each label represents.
- +R (Rules): Specific syntactic or formatting constraints.
- DI (Delusive Instruction): Intentionally misleading instructions used as a control.
2. Benchmarks
The study spanned multiple domains, including sentiment analysis (SST-2, SST-5), natural language inference (MNLI), medical and financial QA (MedQA, FinQA), and structured generative tasks like semantic parsing (MTOP) and question decomposition (Break).
Key Insights
The Flattening Effect
For the majority of classification tasks, the researchers observed a "flattening" of the performance curve. As instructions moved from "No Instruction" to "Detailed," the NICE score rose toward 1.0. In these scenarios, a well-written instruction acts as a stabilizer, making the model nearly indifferent to whether examples are selected via random sampling or sophisticated retrieval methods like BM25.
Schema vs. Content
The study identifies a clear divide between task types. Tasks that remained "Low NICE" (where examples still mattered) were almost exclusively schema-based generative tasks. In tasks like MTOP or NL2BASH, the model must adhere to a rigid, complex output format. The authors argue that while instructions can convey intent, examples are often more efficient at conveying the structural schema required for these outputs.
The Role of Labels
Interestingly, in "High NICE" tasks with detailed instructions, providing incorrect or random labels for the in-context examples did not significantly hurt performance. This suggests that once an instruction provides a clear mapping of the task, the examples serve primarily to demonstrate the expected distribution and output format rather than providing the ground truth logic.
Results and Discussion
- Classification Dominance: On benchmarks like SST-2 and MNLI, NICE scores reached 0.99 to 1.00 with detailed instructions. Randomly sampled examples performed on par with state-of-the-art retrieval methods.
- Generative Sensitivity: Tasks like MTOP and Break exhibited much lower NICE scores (ranging from 0.20 to 0.38). For these tasks, optimized example selection remains a critical component of the pipeline.
- Model Agnosticism: These trends were consistent across a variety of architectures, including GPT-4-Turbo, Llama-2-70B, and Mixtral-8x7B, indicating that the instruction-following capability of the model is the primary driver of the NICE score.
Conclusion
The practical implications of this research are clear: for standard classification and simple reasoning tasks, developers should prioritize writing high-quality, detailed instructions over implementing complex example retrieval systems. However, for tasks involving complex, structured outputs where the "schema" is difficult to describe in natural language, example optimization remains a vital tool for maximizing LLM performance.
Summary updated