Understanding Prompt Sensitivity in Language Models

4,043 concepts2,631 papers100 proposals

Large Language Models (LLMs) frequently demonstrate remarkable capabilities on complex reasoning tasks. Yet, these same models can completely reverse their outputs when faced with semantically irrelevant changes to their inputs. Swapping a single word for a synonym, adding a trailing space, or shuffling the order of multiple-choice options can cause a model's performance to plummet.

This phenomenon—known as prompt sensitivity or instructional brittleness—reveals a fundamental gap between a model's apparent capability and its actual robustness. To build reliable systems, researchers have begun moving beyond simply documenting this fragility, focusing instead on uncovering the underlying mechanics of why it occurs and how it can be measured.

The Spectrum of Brittleness

Prompt sensitivity manifests across nearly every facet of language model evaluation, often driven by minor structural, lexical, or stylistic perturbations:

Why Does Prompt Sensitivity Happen?

To understand why language models are so easily swayed by minor input variations, researchers point to two primary factors: internal decoding confidence and surface-level alignment.

Low Internal Decoding Confidence

In ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs, Jingming Zhuo et al. establish a direct link between a model's internal decoding confidence and its external robustness. When a model has a strong, well-grounded understanding of a task, its token probability distribution has a sharp, decisive peak. In this state, minor prompt variations are not enough to shift the top token.

However, when a model faces a task where its internal confidence is low, the probability distribution flatlines, leaving multiple tokens with similar probabilities. In these scenarios, any minor perturbation—such as a change in persona or formatting—acts as a thumb on the scale, shifting the token probabilities just enough to cross a decision boundary and output a completely different answer.

Surface Alignment vs. Deep Cognitive Traits

A second core reason for this brittleness is that alignment and instruction-tuning often modify surface-level token patterns rather than instilling robust, generalized reasoning.

For instance, when evaluating attitudes, opinions, and values, Bolei Ma et al. observe that models frequently flip their expressed opinions based on the order of answer choices. This lack of robust internal belief systems suggests that subjective evaluations often measure prompt artifacts rather than stable cognitive traits (The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models).

This surface-level behavior is also highly visible in safety alignment. Lanxue Zhang et al. show that minor modifications to system prompts can cause safety scores to drop by as much as 35% (Dynamic Evaluation with Cognitive Reasoning for Multi-turn Safety of Large Language Models). Because the safety behaviors are learned as superficial text patterns rather than deeply integrated ethical principles, slight changes in context easily bypass them.

The Illusion of Few-Shot Mastery

Prompt sensitivity also distorts how we evaluate language model capabilities. In True Few-Shot Learning with Language Models, Ethan Perez et al. expose a major flaw in standard few-shot evaluations. Typically, researchers use large validation sets to experiment with dozens of prompt permutations, example orderings, and hyperparameters, selecting the single configuration that performs best.

In a real-world "true few-shot" setting where no validation data is available, choosing a prompt configuration is akin to random chance. Because model performance is highly volatile depending on the exact wording of a prompt, standard evaluation methods significantly overestimate the real-world utility and stability of few-shot LMs (True Few-Shot Learning with Prompts - A Real-World Perspective).

Turning a Bug into a Feature

While prompt sensitivity is primarily a challenge for deployment, recent work has found ways to leverage this volatility to better understand model behavior.

Quantifying Inconsistency

Instead of relying solely on aggregate correctness scores, which can mask underlying instability, Prakhar Ganesh et al. introduce the concept of "prompt multiplicity" (Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity). By evaluating how often a model changes its individual-level answers across minor prompt variations, prompt multiplicity provides a formal metric to quantify the unpredictable nature of model hallucinations and robustness.

Predicting Accuracy via Stability

If a model's output changes frequently due to minor prompt perturbations, it is highly likely that the model does not actually know the answer. The authors of On the Relation between Sensitivity and Accuracy in In-Context Learning identify a strong negative correlation between prediction stability and accuracy.

[Input Prompt] ───► Perturbation Engine ───► [Variant 1, Variant 2, Variant 3]
                                                       │
                                                       ▼
                                                [LLM Inference]
                                                       │
                                                       ▼
                                            Are outputs identical?
                                             /                \
                                           YES                NO
                                           /                    \
                                  [High Stability]        [Low Stability]
                                         │                       │
                                         ▼                       ▼
                                  Trust Prediction        Abstain / Low Confidence

By intentionally introducing semantically equivalent perturbations (such as paraphrasing instructions or reordering demonstrations) and measuring how much the output fluctuates, their framework, SENSEL, can predict when a model is likely to make an error. This allows the model to safely "abstain" from making a prediction when its outputs are unstable, effectively turning a frustrating behavioral bug into a reliable confidence metric.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 2471 papers
  2. Version 1 · 2026-10-04 · 2522 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_prompt_sensitivity_instructional_brittle_a98b39bb,
  title = {prompt sensitivity (instructional brittleness)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-sensitivity-instructional-brittleness/rdr_a98b39bbdfc14446a6f8302a3dc8acd6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-17},
}

This exact version (version 2, frozen text):

@misc{lacuna_prompt_sensitivity_instructional_brittle_a98b39bb_v2,
  title = {prompt sensitivity (instructional brittleness)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-sensitivity-instructional-brittleness/rdr_a98b39bbdfc14446a6f8302a3dc8acd6/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-17},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake