The Cross-Model Transferability Gap in Prompt Optimization

312 concepts205 papers2 proposals

When we write a prompt for a Large Language Model (LLM), we naturally treat it as a piece of human language—an instruction that any sufficiently intelligent agent fluent in English should understand. If a specific phrasing or a set of few-shot examples boosts accuracy on a reasoning benchmark for one model, it seems intuitive that it should provide a similar benefit to another.

However, a growing body of empirical evidence reveals a starkly different reality: optimized prompts are highly brittle, model-specific interventions. An instruction template or context structure that unlocks peak performance in one model architecture frequently causes performance regressions when transferred to another. This phenomenon is known as the cross-model transferability gap.

Why Prompts Fail to Transfer

To understand why prompts do not generalize, we must move past the illusion that LLMs process language exactly like humans. A prompt is not merely a semantic instruction; it acts as a high-dimensional steering vector navigating a model's unique internal representational space.

As Dongha Lee et al. demonstrate in In Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners, different models possess distinct "internal languages" dictated by their specific token probability distributions. A sequence of logic or a specific phrasing that perfectly aligns with one model's latent weights might be completely misaligned with another's, failing to trigger the intended reasoning paths.

Furthermore, models suffer from unique latent biases inherited from their distinct pre-training histories, model scales, and alignment objectives. Zihan Liang 0001 et al. note in Self-Renewal Prompt Optimizing with Implicit Reasoning that optimizing a prompt using a powerful "teacher" model often yields instructions that are deeply overfitted to that teacher's specific representational space. When applied to a different target architecture, these universal prompts become suboptimal because they ignore the unique behavioral characteristics and sensitivities of the downstream model.

The Illusion of Uniform Benchmarks

This lack of transferability has profound implications for how the AI community evaluates models. Standardized benchmarks typically use static, human-crafted prompts to compare diverse models uniformly. However, this approach introduces a significant evaluation bias.

In PromISe: Releasing the Capabilities of LLMs with Prompt Introspective Search, Minzheng Wang et al. show that there is almost zero overlap in the top-performing prompts across different model families. By forcing all models to use the same prompt, standardized benchmarks artificially bottleneck performance, failing to capture the true maximum capability of diverse architectures.

[Uniform Benchmark Prompt] ───► Model A (Optimal Alignment)   ──► High Score
                            └───► Model B (Misaligned Latent Space) ──► Low Score (Hidden Potential)

To expose these hidden talents, researchers are moving away from static evaluation templates. For instance, Yuxuan Xie et al. introduce TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts, a framework that customizes evaluation instructions to match the unique biases and preferences of each specific architecture, proving that model performance is deeply intertwined with instruction design.

Propagation into RAG and Agentic Workflows

The transferability gap is not confined to simple text prompts; it propagates upward into complex system architectures, such as Retrieval-Augmented Generation (RAG) and multi-agent orchestration.

In RAG systems, intermediate components are often designed to clean, re-rank, or refactor retrieved information before presenting it to the LLM. However, Xinyu Gao et al. in Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation and Zixuan Ke et al. in Bridging the Preference Gap between Retrievers and LLMs identify a distinct "preference gap" between retrievers and generators. Context structures or refactoring strategies optimized to align with one specific large language model's preferences fail to generalize to models of different sizes or architectures, requiring complete retraining or fine-tuning of the bridge components whenever the underlying generator is swapped.

Similarly, the trend toward "agentic" workflows—where models engage in multi-turn debate, self-reflection, or tool-calling—suffers from the same architectural dependency. The authors of When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty find that multi-step reasoning workflows are highly sensitive to the specific model backbone. An orchestration framework or a set of inference-time interventions (such as forcing retries or intercepting tool calls) that enhances one model's performance can actively degrade another's. This occurs because corrective prompts and agentic scaffolds interact unpredictably with the specific decision-making tendencies and baseline calibration profiles of different model families.

The Path Forward: Model-Aware Optimization

Because universal prompts are an illusion, the field is shifting toward automated, model-aware optimization loops. Instead of relying on human intuition or static templates, frameworks like the one proposed by Claudio Spiess et al. in AutoPDL: Automatic Prompt Optimization for LLM Agents treat prompt engineering as a structured machine learning problem.

By placing the target model directly in the optimization loop, these systems use algorithmic search and feedback to discover the precise, often counter-intuitive phrasings and structural patterns required to unlock peak performance for that specific architecture. As models continue to diversify in scale, objective, and training data, prompt engineering must transition from a creative writing exercise into a model-aware optimization science.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_specific_prompt_sensitivity_0254e4b4,
  title = {model-specific prompt sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-specific-prompt-sensitivity/rdr_0254e4b4310e4ecbab2c4c1dba89e937},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_specific_prompt_sensitivity_0254e4b4_v1,
  title = {model-specific prompt sensitivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-specific-prompt-sensitivity/rdr_0254e4b4310e4ecbab2c4c1dba89e937/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake