Can We Predict the Effect of Prompts?

Jae Yong Lee, Sungmin Kang, Shin Yoo

Introduction

Prompt engineering has become a cornerstone of interacting with Large Language Models (LLMs), yet it remains a costly process of trial-and-error. Developers often find themselves running extensive, expensive benchmarks simply to determine which prompt variation—or "vibe"—yields the best results. This "guess-and-check" approach is increasingly unsustainable as models grow in size and complexity.

This paper introduces Predictive Prompt Analysis (PPA), a novel method designed to forecast an LLM's response to a prompt before a single token is generated. By leveraging Sparse Autoencoders (SAEs), the authors developed a prototype called the Syntactic Prevalence Analyzer (SPA). SPA can predict the prevalence of specific code structures with high accuracy while operating significantly faster than traditional benchmarking methods.

Methodology

The core innovation of SPA lies in its use of Sparse Autoencoders to interpret internal model activations. While standard LLM activations are dense and uninterpretable, SAEs decompose them into thousands of sparse, human-understandable "features." For instance, a specific feature might activate only when the model processes Python exception handling logic.

The Syntactic Prevalence Analyzer (SPA) workflow using Sparse Autoencoders to rank candidate prompts based on target feature activation.

The SPA workflow operates in two distinct phases:

  1. Target Feature Extraction: The user defines a syntactic objective (e.g., generating a try-except block). SPA utilizes "positive" and "negative" prompt templates to isolate the SAE features uniquely associated with that goal.
  2. Ranking and Prediction: When presented with multiple candidate prompts, SPA passes them through the LLM's encoder. Rather than generating text, it measures the activation strength of the previously identified target features. Prompts that trigger these features more intensely are ranked higher, as they are more likely to produce the desired output structure.

Experimental Setup

The researchers evaluated SPA using Gemma-2-2b-it in conjunction with Gemma-Scope, a suite of pre-trained SAEs. The evaluation benchmark was the sanitized Mostly Basic Python Problems (MBPP) dataset, comprising 427 programming tasks. The study focused on predicting three specific syntactic structures: try-except clauses, comments, and print statements.

Results and Discussion

SPA demonstrated remarkable predictive power and efficiency. The tool achieved a Pearson correlation of up to 0.994 between its internal predictions and the actual frequency of target structures in the generated output. Furthermore, the analysis was found to be 235x faster than running a full LLM benchmark, taking only 0.4% of the time required for standard inference.

Pearson correlation between SPA predictions and actual model behavior across different layers, highlighting the predictive peak at Layer 16.

A key insight from the study is the identification of a "sweet spot" for prediction within the model's architecture. As shown in the data, features in the middle-to-late layers—specifically Layer 16—were the most predictive of final output. This suggests that these layers are where the model solidifies its "intent" or structural plan for the response. Additionally, the study confirmed that prompt "vibes" (e.g., polite requests vs. direct commands) create distinct, measurable activation patterns that correlate directly with syntactic outcomes.

Conclusion and Future Work

While SPA offers a powerful tool for prompt optimization, it currently faces limitations. It requires access to internal model activations and high-quality pre-trained SAEs, which are primarily available for open-weight models. Future research may explore "adversarial transferability," using open models to predict the behavior of closed-source systems like GPT-4.

Furthermore, while SPA excels at predicting syntactic structures, the next major challenge is moving toward semantic prediction—forecasting the logical correctness of a model's response rather than just its formatting. Despite these hurdles, PPA represents a significant step toward making prompt engineering a more scientific and computationally efficient discipline.

Summary updated

BibTeX
@inproceedings{lee2025can,
  author = {Jae Yong Lee and Sungmin Kang and Shin Yoo},
  title = {{Can We Predict the Effect of Prompts?}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2025},
  doi = {10.1145/3696630.3728516},
}
Figure