Multi-Prompt Ensembling for Robust Inference

28 concepts22 papers1 proposal

Large Language Models (LLMs) are notoriously brittle. A minor change in a prompt’s wording, the choice of a few-shot example, or even the order of demonstrations can lead to wildly different outputs. As Yang et al. observed, while models can be "nudged" into providing valid answers, their performance is often inconsistent and highly sensitive to specific phrasing.

Multi-prompt ensembling addresses this instability by querying a model with multiple variations of a task and aggregating the results. Instead of searching for a single "perfect" prompt, this approach treats the model’s response to a single prompt as a noisy sample, using an ensemble to smooth out variance and uncover a more robust underlying prediction.

The Sensitivity Problem

The core motivation for ensembling is the high variance inherent in In-Context Learning (ICL). Voronov et al. demonstrated that the "best" prompt templates rarely transfer between different models or methods, making evaluation a moving target. Similarly, Cao et al. noted that models are highly sensitive to the specific choice and order of in-context demonstrations.

This sensitivity creates a "prompt engineering" bottleneck where researchers must curate large validation sets just to identify which wording works best. Schick and Schuetze argue that ensembling multiple prompts mitigates this need, enabling "true" few-shot learning where models perform reliably without extensive prompt tuning on labeled data.

Mechanisms of Ensembling

Research in this area generally follows three axes of variation to create an ensemble:

  1. Template Variation: The authors of Pattern-Exploiting Training (PET) reformulate tasks into multiple "fill-in-the-blank" patterns (e.g., "It was [MASK]" vs "The movie was [MASK]"). By aggregating predictions across these diverse linguistic phrasings, the system becomes less dependent on any single suboptimal template.
  2. Demonstration Sampling: Rather than picking one fixed set of few-shot examples, Yao et al. proposed In-Context Sampling (ICS). This technique uses a "committee" of prompts, each containing a unique combination of task examples, to leverage the diverse implicit guidance provided by different subsets of data.
  3. Input Perturbation: For complex evaluation tasks, such as text-to-3D generation, Wu et al. suggest generating slightly perturbed inputs and aggregating the outputs to achieve more stable and human-aligned results.

Aggregation and Synthesis

Once multiple prompts are processed, the system must synthesize the results. The most common method is simple averaging of log probabilities or scores, which Ostheimer et al. found consistently yields higher correlations with human judgment than any single prompt.

However, more sophisticated methods are emerging. Cheng et al. introduced the Multi-Prompting Decoder (MPD), which queries a model with several prompts simultaneously and uses Optimal Transport to synthesize the resulting internal representations. This allows the model to aggregate richer knowledge from its hidden states rather than just its final output tokens.

Applications and Benefits

Ensembling has proven effective across diverse domains:

  • Quality Estimation: Eo et al. used template ensembling to win the WMT22 Critical Error Detection task, stabilizing scores for translation quality.
  • Software Engineering: Wang et al. applied multi-template prompt-tuning to link software issues to code commits, achieving high F1-scores by reducing sensitivity to specific commit message wordings.
  • Fact Verification: Strelec et al. and Baek et al. both utilized ensembles to stabilize semantic reasoning and error detection in knowledge-heavy tasks.
  • Security: Zhang et al. integrated ensembling with Fully Homomorphic Encryption to ensure that LLM outputs remain robust against adversarial manipulations while maintaining privacy.

By shifting the focus from finding the best prompt to aggregating a distribution of prompts, this research direction provides a path toward more predictable and reliable AI systems that are less prone to the "brittleness" of natural language instructions.

Key Foundational Work

Prompting for Multimodal Hateful Meme Classification (2022) — contributed 2 concept elements to this direction.
KU X Upstage's Submission for the WMT22 Quality Estimation: Critical Error Detection Shared Task (2022) — contributed 1 concept elements to this direction.
What GPT Knows About Who is Who (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_prompt_ensembling_020b39cf,
  title = {multi-prompt ensembling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-prompt-ensembling/rdr_020b39cfb8bf4d6683135971b92cab7b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_prompt_ensembling_020b39cf_v1,
  title = {multi-prompt ensembling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-prompt-ensembling/rdr_020b39cfb8bf4d6683135971b92cab7b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake