Diverse Exemplar Selection for In-Context Learning

41 concepts33 papers1 proposal

When prompting a Large Language Model (LLM) using In-Context Learning (ICL), the few-shot examples you provide act as a localized tutorial. The standard approach to building this tutorial is intuitive: find the examples in your training data that are most semantically similar to the user's query, typically via $k$-Nearest Neighbors.

However, selecting exemplars based solely on proximity creates a critical failure mode: redundancy. If a query requires multi-step reasoning, and the top four retrieved examples all demonstrate the exact same logical sub-step or linguistic pattern, the model learns very little about the broader boundaries of the task. As Soumya Suvra Ghosal et al. point out in their work on KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning, a lack of diversity prevents the model from understanding the variations and edge cases of the target task.

To build a robust context, we must shift from picking the individually most relevant examples to picking a collectively optimal set. This is the core of diversity-aware exemplar selection: balancing relevance to the query with diversity among the demonstrations.

The Redundancy Trap and Why Proximity Fails

Why does pure similarity hurt performance? When an LLM is exposed to highly repetitive context, it tends to overfit to specific linguistic or structural patterns. For instance, in knowledge-grounded dialog, Qinyu Chen et al. observed in Exploring In-Context Learning for Knowledge Grounded Dialog Generation that low-diversity prompts cause models to converge on a single response pattern, limiting their ability to faithfully apply external facts. Similarly, in multimodal settings, Yang Luo et al. demonstrated in How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? that standard retrieval methods introduce severe redundancy, missing critical task variations.

Surprisingly, the negative impact of redundancy is so pronounced that even randomly selecting examples can sometimes outperform similarity-based retrieval. In HW-TSC at Multilingual Counterspeech Generation, Hao Yang et al. showed that random selection provides a naturally diverse range of contexts, preventing the model from overfitting to narrow linguistic styles and helping it better capture the underlying task logic. The goal, then, is to formalize this diversity without losing relevance.

Algorithmic Frameworks for Balancing Relevance and Diversity

To systematically prevent redundancy, researchers employ algorithms that explicitly trade off a candidate example's relevance to the query against its similarity to already-chosen examples. Two primary frameworks dominate this space: Maximal Marginal Relevance and Determinantal Point Processes.

Maximal Marginal Relevance (MMR)

MMR is a greedy selection strategy. Instead of picking the top-$k$ closest items, it selects the next exemplar by maximizing a weighted combination of relevance to the user query and dissimilarity to the exemplars already in the prompt window.

$$\text{MMR} = \arg\max_{D_i \in R \setminus S} \left[ \lambda \cdot \text{Sim}(D_i, Q) - (1 - \lambda) \cdot \max_{D_j \in S} \text{Sim}(D_i, D_j) \right]$$

This approach has proven highly effective across diverse domains:

Determinantal Point Processes (DPP)

While MMR is greedy, Determinantal Point Processes (DPPs) offer an elegant, global probabilistic framework for subset selection. A DPP defines a probability distribution over subsets of data points, where the probability of selecting a specific subset is proportional to the determinant of a similarity kernel matrix.

Geometrically, this determinant corresponds to the volume of the parallelotope spanned by the feature vectors of the selected items. If two examples are highly similar, their vectors point in nearly the same direction, squashing the volume (and the probability) to near zero.

As surveyed by Man Luo et al. in In-context Learning with Retrieved Demonstrations for Language Models: A Survey, DPPs are widely used to ensure retrieved examples cover a broad range of a task's reasoning space. For example, Yanan Cao et al. introduced TISE: A Tripartite In-context Selection Method for Event Argument Extraction, which uses a DPP to balance semantic similarity, diversity, and event-specific correlations, proving that a smaller set of high-variety examples easily outperforms a larger, redundant set. Similarly, the authors of PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from related Example Banks leveraged DPPs to select diverse cross-lingual exemplars from high-resource languages to boost performance on low-resource Indic languages.

Multi-Faceted and Interaction-Driven Diversity

Diversity does not only mean spreading out points in a static embedding space. Recent work looks at diversity across multiple dimensions of a task:

Why It Works: The Mechanics of Generalization

Ultimately, why does a diverse prompt make an LLM smarter?

  1. Compositional Generalization: To master compositional concepts (like combining "red" and "chair"), models need to see varied contexts. Guangyue Xu et al. showed in MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition that redundancy in retrieved context actively hinders an agent's ability to synthesize novel concepts from memory.
  2. Broad Reasoning Coverage: Instead of optimizing for the single closest match, treating the prompt as a collective portfolio of reasoning patterns (Sameer Singh et al. in Leveraging In-Context Learning for Language Model Agents) or using multiple diverse prompts via In-Context Sampling (Bingsheng Yao et al. in More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling) ensures that the model can handle a wide array of sub-tasks and logical variations at test time.
  3. Mitigating Generation Bias: Even outside of direct prompting, injecting diversity into intermediate generation steps—such as the counterspeech pipeline proposed by Wanzheng Zhu in Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech—directly improves the final output's relevance by avoiding generic, low-diversity traps.

Key Foundational Work

Complementary Explanations for Effective In-Context Learning (2023) — contributed 2 concept elements to this direction.
Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking (2023) — contributed 2 concept elements to this direction.
Pool-Search-Demonstrate: Improving Data-wrangling LLMs via better in-context examples (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prompt_diversity_d70e1cff,
  title = {prompt diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-diversity/rdr_d70e1cfff32d4916903f34fd3063cc5b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_prompt_diversity_d70e1cff_v1,
  title = {prompt diversity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-diversity/rdr_d70e1cfff32d4916903f34fd3063cc5b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake