When prompting a Large Language Model (LLM) using In-Context Learning (ICL), the few-shot examples you provide act as a localized tutorial. The standard approach to building this tutorial is intuitive: find the examples in your training data that are most semantically similar to the user's query, typically via $k$-Nearest Neighbors.
However, selecting exemplars based solely on proximity creates a critical failure mode: redundancy. If a query requires multi-step reasoning, and the top four retrieved examples all demonstrate the exact same logical sub-step or linguistic pattern, the model learns very little about the broader boundaries of the task. As Soumya Suvra Ghosal et al. point out in their work on KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning, a lack of diversity prevents the model from understanding the variations and edge cases of the target task.
To build a robust context, we must shift from picking the individually most relevant examples to picking a collectively optimal set. This is the core of diversity-aware exemplar selection: balancing relevance to the query with diversity among the demonstrations.
The Redundancy Trap and Why Proximity Fails
Why does pure similarity hurt performance? When an LLM is exposed to highly repetitive context, it tends to overfit to specific linguistic or structural patterns. For instance, in knowledge-grounded dialog, Qinyu Chen et al. observed in Exploring In-Context Learning for Knowledge Grounded Dialog Generation that low-diversity prompts cause models to converge on a single response pattern, limiting their ability to faithfully apply external facts. Similarly, in multimodal settings, Yang Luo et al. demonstrated in How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? that standard retrieval methods introduce severe redundancy, missing critical task variations.
Surprisingly, the negative impact of redundancy is so pronounced that even randomly selecting examples can sometimes outperform similarity-based retrieval. In HW-TSC at Multilingual Counterspeech Generation, Hao Yang et al. showed that random selection provides a naturally diverse range of contexts, preventing the model from overfitting to narrow linguistic styles and helping it better capture the underlying task logic. The goal, then, is to formalize this diversity without losing relevance.
Algorithmic Frameworks for Balancing Relevance and Diversity
To systematically prevent redundancy, researchers employ algorithms that explicitly trade off a candidate example's relevance to the query against its similarity to already-chosen examples. Two primary frameworks dominate this space: Maximal Marginal Relevance and Determinantal Point Processes.
Maximal Marginal Relevance (MMR)
MMR is a greedy selection strategy. Instead of picking the top-$k$ closest items, it selects the next exemplar by maximizing a weighted combination of relevance to the user query and dissimilarity to the exemplars already in the prompt window.
$$\text{MMR} = \arg\max_{D_i \in R \setminus S} \left[ \lambda \cdot \text{Sim}(D_i, Q) - (1 - \lambda) \cdot \max_{D_j \in S} \text{Sim}(D_i, D_j) \right]$$
This approach has proven highly effective across diverse domains:
- Complex Reasoning: Xi Ye et al. utilized MMR in Complementary Explanations for Effective In-Context Learning to assemble complementary reasoning paths, ensuring the prompt acts as a comprehensive tutorial rather than a repetitive loop.
- Medical Summarization: In SummQA at MEDIQA-Chat 2023: In-Context Learning with GPT-4 for Medical Summarization, Yash Mathur et al. used MMR to expose the model to a wider array of clinical patterns within the few-shot context.
- Action Anticipation: It was applied to long-term action anticipation in egocentric video by Sanghwan Kim et al. in PALM: Predicting Actions through Language Models to prevent overfitting to narrow subsets of human activities.
- Reinforcement Learning: In Modelling the control of offline processing with reinforcement learning, Eleanor Spens et al. integrated MMR to maintain a diverse training/replay set, ensuring broad coverage of a state space during offline processing.
Determinantal Point Processes (DPP)
While MMR is greedy, Determinantal Point Processes (DPPs) offer an elegant, global probabilistic framework for subset selection. A DPP defines a probability distribution over subsets of data points, where the probability of selecting a specific subset is proportional to the determinant of a similarity kernel matrix.
Geometrically, this determinant corresponds to the volume of the parallelotope spanned by the feature vectors of the selected items. If two examples are highly similar, their vectors point in nearly the same direction, squashing the volume (and the probability) to near zero.
As surveyed by Man Luo et al. in In-context Learning with Retrieved Demonstrations for Language Models: A Survey, DPPs are widely used to ensure retrieved examples cover a broad range of a task's reasoning space. For example, Yanan Cao et al. introduced TISE: A Tripartite In-context Selection Method for Event Argument Extraction, which uses a DPP to balance semantic similarity, diversity, and event-specific correlations, proving that a smaller set of high-variety examples easily outperforms a larger, redundant set. Similarly, the authors of PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from related Example Banks leveraged DPPs to select diverse cross-lingual exemplars from high-resource languages to boost performance on low-resource Indic languages.
Multi-Faceted and Interaction-Driven Diversity
Diversity does not only mean spreading out points in a static embedding space. Recent work looks at diversity across multiple dimensions of a task:
- Structural and Categorical Variation: In Schema and Natural Language Aware In-Context Learning for Improved GraphQL Query Generation, Nitin Gupta 0005 et al. developed the SNAIL framework, which rotates between structural, categorical, and semantic metrics to ensure the model receives a balanced variety of schema skeletons rather than just textually similar examples.
- Interaction over Individual Quality: In EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning, Kiran Purohit et al. argue that ICL performance is driven more by the interaction between examples in a prompt than by the individual accuracy of each example. A diverse subset that covers a broad range of task-relevant concepts collectively outperforms sets of individually high-quality but redundant examples.
- Relational and Prompt-Level Diversity: Yanbin Wei et al. applied diversity-based ordering to relational analogies in KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion to show models a broad range of contexts. To prevent prompt selection mechanisms from collapsing into a single choice, Qiushi Huang et al. introduced a contrastive learning objective in Selective Prompting Tuning for Personalized Conversations with LLMs that forces the system to utilize a wide variety of personalized prompts.
Why It Works: The Mechanics of Generalization
Ultimately, why does a diverse prompt make an LLM smarter?
- Compositional Generalization: To master compositional concepts (like combining "red" and "chair"), models need to see varied contexts. Guangyue Xu et al. showed in MetaReVision: Meta-Learning with Retrieval for Visually Grounded Compositional Concept Acquisition that redundancy in retrieved context actively hinders an agent's ability to synthesize novel concepts from memory.
- Broad Reasoning Coverage: Instead of optimizing for the single closest match, treating the prompt as a collective portfolio of reasoning patterns (Sameer Singh et al. in Leveraging In-Context Learning for Language Model Agents) or using multiple diverse prompts via In-Context Sampling (Bingsheng Yao et al. in More Samples or More Prompts? Exploring Effective Few-Shot In-Context Learning for LLMs with In-Context Sampling) ensures that the model can handle a wide array of sub-tasks and logical variations at test time.
- Mitigating Generation Bias: Even outside of direct prompting, injecting diversity into intermediate generation steps—such as the counterspeech pipeline proposed by Wanzheng Zhu in Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech—directly improves the final output's relevance by avoiding generic, low-diversity traps.
Key Foundational Work
Complementary Explanations for Effective In-Context Learning (2023) — contributed 2 concept elements to this direction.
Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking (2023) — contributed 2 concept elements to this direction.
Pool-Search-Demonstrate: Improving Data-wrangling LLMs via better in-context examples (2023) — contributed 2 concept elements to this direction.