Dynamic Demonstration Retrieval for In-Context Learning

24 concepts23 papers2 proposals

In-context learning (ICL) allows large language models (LLMs) to execute complex tasks simply by observing a few examples appended to the prompt. Traditionally, these few-shot examples are static—a fixed set of hand-picked demonstrations used for every single incoming query. However, a static prompt is a "one-size-fits-all" solution that frequently fails when queries are highly diverse, structurally intricate, or out-of-distribution.

To overcome this, a prominent research direction treats example selection as a dynamic retrieval problem. For any given input query, a retrieval mechanism searches an annotated data pool to find the most relevant demonstrations and injects them into the context window on the fly. This paradigm bridges the gap between massive parametric models and non-parametric memory, allowing models to reason by analogy using contextually tailored precedents.

The Intuition: Why Local Context Matters

Why does selecting similar examples drastically improve performance compared to using random or static ones? Large language models are highly sensitive to the distribution, syntax, and formatting of their prompts. When a model is given a static set of examples, it must perform two difficult steps simultaneously: infer the abstract task mapping and filter out irrelevant surface features that do not apply to the current query.

Dynamic retrieval provides the model with local context. As shown by Gao et al. in their LM-BFF framework, incorporating semantically similar demonstrations yields massive performance gains over standard fine-tuning for moderately sized language models. When the demonstrations share structural or semantic features with the query, the model does not need to make massive cognitive leaps. Instead, it can directly copy the syntactic templates or logical steps demonstrated in the prompt.

Luo et al. formalize this as query-conditioned demonstration selection, proving that even instruction-tuned models—which have already seen the underlying training data during alignment—benefit significantly from having contextually relevant patterns surfaced immediately prior to generation.

Mechanisms: Lexical vs. Semantic Similarity

To retrieve the best examples, systems must define what makes an example "similar" to a query. Generally, this selection relies on two distinct retrieval mechanisms, as highlighted by Geng et al.:

  1. Lexical Retrieval: Focuses on token overlap (e.g., BM25). This is highly effective for tasks where exact keywords, variable names, or specific surface syntax dictate the correct output.
  2. Semantic Retrieval: Uses dense vector embeddings to capture abstract meaning, intent, or functional behavior.

For instance, Zhang utilizes dense vector embeddings to retrieve question-query pairs that align closely with the structural intent of a user's query across diverse databases. Similarly, Amplayo et al. use semantic similarity metrics to retrieve exemplars that match the specific facets of complex, long-form questions, ensuring the model's intermediate reasoning steps are grounded in relevant templates. For machine translation, Peng et al. demonstrate that similarity-based example selection provides highly relevant lexical and syntactic templates to guide the generation process.

Handling Complex Reasoning and Structure

Dynamic retrieval is particularly crucial for tasks involving strict structural constraints, such as code generation, mathematical reasoning, and semantic parsing.

In mathematical and numerical reasoning, a single static prompt cannot cover the vast variety of word problem structures. Jie et al. demonstrate that dynamically retrieving examples based on semantic similarity provides models with tailored structural templates, allowing them to map complex word problems to Python programs step-by-step.

For structured tasks like semantic parsing and text-to-query translation, syntax matters immensely. Shin et al. show that dynamic prompting provides pertinent demonstrations of mapping natural language to grammar-constrained sublanguages. When translating medical text into exotic query languages like Cypher or SPARQL, Fürst et al. find that retrieving semantically similar examples provides the necessary structural and syntactic precedents that models otherwise struggle to generate. Furthermore, Madaan et al. leverage this in their COCOGEN method, populating prompts with examples sharing high structural similarity to guide models through complex event graph generation framed as code completion.

This approach also helps models handle "hard" examples. Saha et al. note that while language models struggle to explain difficult reasoning tasks, pulling examples from a pool based on similarity to the target test case ensures that the logic required for specific, difficult edge cases is explicitly demonstrated.

Generalization Across Domains and Modalities

The utility of similarity-based prompt retrieval extends far beyond general natural language processing into specialized industrial applications, tabular data, and even non-textual modalities.

  • Content Moderation & Finance: In content moderation, Wang et al. use retrieval-augmented few-shot prompting to inject immediate precedents for specific moderation logic, allowing smaller models to generalize to completely unseen moderation categories. In finance, Liu et al. employ embedding-based retrieval to help models adapt their reasoning to the subtle nuances of financial misinformation claims.
  • E-Commerce & Marketing: Sun et al. use sentence embeddings to retrieve historical user sessions, providing contextually aligned examples that help models infer user intent and generate personalized product bundles. For abstract marketing demands, Wang et al. show that context relevance via similarity retrieval is far more critical than example quantity for guiding analogical reasoning.
  • Tabular Data: Moving away from text entirely, Ma et al. introduce TabDPT, a tabular foundation model that uses a retrieval mechanism to extract the most similar rows from a dataset. This provides local context during inference, enabling the model to leverage local data distributions without any weight fine-tuning.
  • Speech & Continual Learning: The paradigm even applies to continuous representations. Li et al. implement a dynamic selection mechanism for speaker verification, retrieving prompt embeddings from a learnable pool based on feature similarity. In online continual learning where labels are delayed, the authors use feature embeddings of unlabeled data to query a memory buffer, dynamically adjusting the model's training focus toward the most relevant past experiences.

Key Foundational Work

Language Models of Code are Few-Shot Commonsense Learners (2022) — contributed 1 concept elements to this direction.
Constrained Language Models Yield Few-Shot Semantic Parsers (2021) — contributed 1 concept elements to this direction.
Making Pre-trained Language Models Better Few-shot Learners (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_similarity_based_prompt_retrieval_8e02ef77,
  title = {similarity-based prompt retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-prompt-retrieval/rdr_8e02ef77f55b45c187f553edabefb467},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_similarity_based_prompt_retrieval_8e02ef77_v1,
  title = {similarity-based prompt retrieval},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/similarity-based-prompt-retrieval/rdr_8e02ef77f55b45c187f553edabefb467/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake