Prompt Performance Prediction for Generative IR

Nicolas Bizzozzero, Ihab Bendidi, Olivier Risser-Maroix

Introduction

Generative AI has democratized digital creation, yet the process remains largely iterative and computationally expensive. Users often engage in a "trial and error" cycle of prompt engineering to achieve desired results. This research introduces Prompt Performance Prediction (PPP), a task analogous to Query Performance Prediction in traditional Information Retrieval. PPP aims to estimate the quality of a generated image—such as its aesthetic appeal, memorability, or composition—using only the input text prompt, thereby optimizing the generative workflow before a single GPU cycle is spent on image creation.

Methodology

The researchers frame PPP as a regression task: given a text prompt, the goal is to predict a performance score. The approach involves several key steps:

  1. Feature Extraction: Utilizing eight different pre-trained models (including CLIP, T5, and GPT-2) to convert text prompts into mathematical embeddings.
  2. Linear Probing: Applying a simple, efficient linear regressor on top of these frozen embeddings to maintain a "plug-and-play" architecture.
  3. Ground Truth Generation: Since large-scale human-rated prompt datasets are scarce, the authors used a proxy method. Existing prompt-image pairs were processed through state-of-the-art image graders to generate labels for the corresponding text prompts.
The Prompt Performance Prediction (PPP) pipeline illustrating how prompt components are used to predict a performance score.

The Modality Gap and Predictability

A core challenge identified in the study is the "modality gap." While models like CLIP are designed to align text and images, they do not occupy identical regions in the embedding space. PCA analysis reveals a distinct separation between prompt and image embeddings, necessitating a dedicated PPP layer to bridge this gap rather than simply applying image-based predictors to text.

PCA visualization showing the clear separation (modality gap) between image and prompt embeddings.

Furthermore, the research highlights that predictability is influenced by prompt structure. Prompts in datasets like Midjourney and Stable Diffusion often contain numerous "modifiers" (e.g., "4k", "unreal engine"), which reduce stochasticity and make final quality more predictable compared to the simpler, shorter prompts typically found in DALL-E 2 galleries.

Key Results

  • Model Performance: CLIP-ViT-L-14 emerged as the superior feature extractor, achieving Pearson correlation scores as high as 0.83 for memorability on the Midjourney dataset. Pure language models like sentence-t5-xl were also highly competitive.
  • Human Alignment: In experiments involving real paintings, the text-based model predicted human appreciation with a correlation of ~0.65, which is comparable to typical human-to-human agreement levels (0.60–0.82).
  • Content vs. Geometry: The study found it is significantly easier to predict memorability (tied to the subject matter) than composition (tied to geometric pixel arrangement) from a text prompt.

Conclusion and Future Work

PPP offers a path toward more efficient Generative IR by reducing wasted compute and assisting users in prompt optimization. Future research directions include automatic prompt reformulation to "boost" predicted scores, bias detection in aesthetic scoring, and the development of more complex predictive architectures beyond linear probes.

Summary updated

Abstract

The ability to predict the performance of a query before results are returned has been a longstanding challenge in Information Retrieval (IR) systems. Inspired by this task, we introduce, in this paper, a novel task called “Prompt Performance Prediction” that aims to predict the performance of a prompt, before obtaining the actual generated images. We demonstrate the plausibility of our task by measuring the correlation coefficient between predicted and actual performance scores across: three datasets containing pairs of prompts and generated images and three art domain datasets of real images and real user appreciation ratings. Our results show promising performance prediction capabilities, suggesting potential applications for optimizing user prompts.
BibTeX
@inproceedings{bizzozzero2024prompt,
  author = {Nicolas Bizzozzero and Ihab Bendidi and Olivier Risser-Maroix},
  title = {{Prompt Performance Prediction for Generative IR}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647937},
}
Figure