Autonomous Prompt Generation for In-Context Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 127 papers27 conceptsfrozen 2026-10-04

The performance of Large Language Models (LLMs) often hinges on the quality of the "context" provided to them. Traditionally, this has meant a human expert must meticulously hand-craft a few examples—known as few-shot demonstrations—or write out step-by-step reasoning chains (Chain-of-Thought) to guide the model.

However, this manual process creates a significant scalability bottleneck. As Furu Wei and colleagues point out in Auto-ICL, the "human-in-the-loop" requirement limits our ability to deploy models across thousands of niche tasks. A new research direction seeks to eliminate this bottleneck by allowing LLMs to autonomously generate their own demonstrations, instructions, and reasoning structures.

The Human Bottleneck in Reasoning

To solve complex problems, models often need more than just a question; they need a template for how to think. In tasks like Text-to-SQL, Hanchong Zhang and the ACT-SQL team argue that manual authoring of these reasoning sequences is too labor-intensive to scale.

Even when humans do provide these examples, they may introduce bias or fail to generalize. Kaya Stechly and others in Chain of Thoughtlessness? suggest that human-engineered traces can sometimes lead models into narrow pattern matching rather than true algorithmic learning. If the model can instead "teach itself," it might find reasoning paths that are more native to its own internal representations.

LLMs as Their Own Teachers

The core intuition behind this research is that an LLM's latent knowledge is often broader than its zero-shot performance suggests. By asking the model to "contemplate" or "recall" similar problems before solving a new one, we can move that latent knowledge into the active context window.

In Self-Contemplation Prompting (SEC), Rui Li and Guoyin Wang demonstrate that models like GPT-4 are now competent enough to generate their own few-shot demonstrations. This "self-teaching" matches or exceeds the performance of human-assisted learning because the model generates examples that are perfectly tailored to the complexity it is capable of handling.

Similarly, Michihiro Yasunaga and Percy Liang introduced Analogical Prompting in Large Language Models as Analogical Reasoners. This technique mimics human cognition: when faced with a new math problem, a human might first think, "What similar problems have I solved before?" The model is prompted to self-generate relevant examples and high-level knowledge on the fly before tackling the target problem, effectively creating its own custom training data for every query.

Automating the Reasoning Chain

Beyond just generating examples, researchers are automating the "Chain-of-Thought" (CoT) itself. Zhuosheng Zhang and Aston Zhang developed Auto-CoT, which uses a two-step process:

  1. Clustering: It groups diverse questions from a dataset to ensure a broad range of logic is covered.
  2. Generation: It uses the model to generate reasoning chains for a representative question in each cluster.

This ensures that the model isn't just seeing one type of logic, but a diverse set of "reasoning recipes." Zhihong Shao and Nan Duan take this further in Synthetic Prompting by alternating between generating reasoning chains and matching questions, then selecting only the most complex and diverse examples to keep the model's "self-training" high-quality.

Discovering Optimal Reasoning Structures

The most recent evolution in this field moves away from fixed prompts toward dynamic "reasoning structures." Instead of asking a model to follow a specific path like "Chain-of-Thought," researchers are letting the model choose its own strategy.

In SELF-DISCOVER, Pei Zhou describes a framework where LLMs self-compose reasoning structures by selecting from a toolkit of strategies (like "critical thinking" or "step-back abstraction"). This allows the model to adapt its logic to the specific problem at hand without human intervention.

To refine these strategies, Weijia Xu and Nebojsa Jojic proposed Reprompting, an iterative algorithm that treats prompt discovery as a search problem. By using Gibbs sampling to evolve "reasoning recipes," the system discovers prompts that outperform human-written ones by significant margins.

Why It Works: The Latent-to-Explicit Shift

The success of these methods suggests that the primary hurdle for LLMs is not a lack of knowledge, but a lack of focus. Zero-shot prompting asks the model to jump directly from a question to an answer, which is a high-entropy leap.

By generating its own examples (Auto-ICL), recalling analogies (Analogical Prompting), or building its own reasoning datasets (Self-prompted CoT), the model is essentially performing a "warm-up" exercise. It moves relevant concepts from its massive parameters into its immediate context, reducing the complexity of the final step and allowing it to solve problems that were previously out of reach.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_autonomous_prompt_generation_for_in_cont_81159292,
  title = {Autonomous Prompt Generation for In-Context Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autonomous-prompt-generation-for-in-context-learning/rdr_81159292d8564bd7913ca09c785557ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_autonomous_prompt_generation_for_in_cont_81159292_v1,
  title = {Autonomous Prompt Generation for In-Context Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autonomous-prompt-generation-for-in-context-learning/rdr_81159292d8564bd7913ca09c785557ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}