NICE: To Optimize In-Context Examples or Not?

Pragya Srivastava, Satvik Golechha, Amit Deshpande, Amit Sharma

Abstract

In-Context Learning (ICL) has traditionally relied on the careful selection of few-shot examples to maximize Large Language Model (LLM) performance. However, the rise of instruction-tuned models raises a critical question: does the quality of the instruction render example optimization obsolete? This paper introduces NICE (Normalized Invariability to Choice of Examples), a metric designed to quantify a model's sensitivity to example selection. Our findings reveal a "flattening effect": as instructions become more detailed, the performance variance across different example sets vanishes for classification tasks. Conversely, complex generative tasks maintain a high sensitivity to example choice, suggesting that the necessity of example optimization is task-dependent.

Introduction

In the evolving landscape of In-Context Learning, a significant amount of research has been dedicated to "example selection"—the process of retrieving the most relevant demonstrations from a training set to include in a prompt. While effective, these retrieval algorithms add significant computational overhead. This paper investigates whether this effort is still justified for modern, instruction-tuned LLMs. By systematically varying instruction detail and measuring performance stability via the NICE metric, we provide a framework for deciding when to optimize examples and when to focus solely on prompt engineering.

The NICE Metric

To measure how much example choice actually matters, the authors propose the Normalized Invariability to Choice of Examples (NICE). The metric is calculated by partitioning candidate examples into "bins" based on their cosine similarity to the query and then comparing the average performance across all bins to the peak performance of the best-performing bin:

[ NICE = \frac{\text{Average Performance}}{\text{Maximum Performance}} ]

  • High NICE (approaching 1.0): The model is robust; performance is consistent regardless of which examples are chosen.
  • Low NICE (approaching 0.0): The model is highly sensitive; picking the "wrong" examples leads to significant performance degradation.

Experimental Methodology

1. Instruction Hierarchy

The researchers tested models across a spectrum of instruction quality, ranging from no instruction to highly specific rules:

  • NI (No Instruction): Only few-shot examples are provided.
  • TD (Task Definition): A concise one-sentence description of the task.
  • +LS (Label Space): Inclusion of valid output labels (e.g., "Options: Positive, Negative").
  • +LM (Label Meanings): Definitions of what each label represents.
  • +R (Rules): Specific syntactic or formatting constraints.
  • DI (Delusive Instruction): Intentionally misleading instructions used as a control.

2. Benchmarks

The study spanned multiple domains, including sentiment analysis (SST-2, SST-5), natural language inference (MNLI), medical and financial QA (MedQA, FinQA), and structured generative tasks like semantic parsing (MTOP) and question decomposition (Break).

Key Insights

The Flattening Effect

For the majority of classification tasks, the researchers observed a "flattening" of the performance curve. As instructions moved from "No Instruction" to "Detailed," the NICE score rose toward 1.0. In these scenarios, a well-written instruction acts as a stabilizer, making the model nearly indifferent to whether examples are selected via random sampling or sophisticated retrieval methods like BM25.

Schema vs. Content

The study identifies a clear divide between task types. Tasks that remained "Low NICE" (where examples still mattered) were almost exclusively schema-based generative tasks. In tasks like MTOP or NL2BASH, the model must adhere to a rigid, complex output format. The authors argue that while instructions can convey intent, examples are often more efficient at conveying the structural schema required for these outputs.

The Role of Labels

Interestingly, in "High NICE" tasks with detailed instructions, providing incorrect or random labels for the in-context examples did not significantly hurt performance. This suggests that once an instruction provides a clear mapping of the task, the examples serve primarily to demonstrate the expected distribution and output format rather than providing the ground truth logic.

Results and Discussion

  • Classification Dominance: On benchmarks like SST-2 and MNLI, NICE scores reached 0.99 to 1.00 with detailed instructions. Randomly sampled examples performed on par with state-of-the-art retrieval methods.
  • Generative Sensitivity: Tasks like MTOP and Break exhibited much lower NICE scores (ranging from 0.20 to 0.38). For these tasks, optimized example selection remains a critical component of the pipeline.
  • Model Agnosticism: These trends were consistent across a variety of architectures, including GPT-4-Turbo, Llama-2-70B, and Mixtral-8x7B, indicating that the instruction-following capability of the model is the primary driver of the NICE score.

Conclusion

The practical implications of this research are clear: for standard classification and simple reasoning tasks, developers should prioritize writing high-quality, detailed instructions over implementing complex example retrieval systems. However, for tasks involving complex, structured outputs where the "schema" is difficult to describe in natural language, example optimization remains a vital tool for maximizing LLM performance.

Summary updated

Abstract

Recent work shows that in-context learning and optimization of in-context examples (ICE) can significantly improve the accuracy of large language models (LLMs) on a wide range of tasks, leading to an apparent consensus that ICE optimization is crucial for better performance. However, most of these studies assume a fixed or no instruction provided in the prompt. We challenge this consensus by investigating the necessity of optimizing ICE when task-specific instructions are provided and find that there are many tasks for which it yields diminishing returns. In particular, using a diverse set of tasks and a systematically created instruction set with gradually added details, we find that as the prompt instruction becomes more detailed, the returns on ICE optimization diminish. To characterize this behavior, we introduce a task-specific metric called Normalized Invariability to Choice of Examples (NICE) that quantifies the learnability of tasks from a given instruction, and provides a heuristic to help decide whether to optimize instructions or ICE for a new task. Given a task, the proposed metric can reliably predict the utility of optimizing ICE compared to using random ICE. Our code is available at https://github.com/microsoft/nice-icl.
BibTeX
@inproceedings{srivastava2024nice,
  author = {Pragya Srivastava and Satvik Golechha and Amit Deshpande and Amit Sharma},
  title = {{NICE: To Optimize In-Context Examples or Not?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.300},
}
Figure