PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction

Eduard Poesina, Adriana Valentina Costache, Adrian-Gabriel Chifu, Josiane Mothe, Radu Tudor IonescuCVPR 2025 · DOI

Introduction

In the field of Information Retrieval (IR), Query Performance Prediction (QPP) is a well-established technique for estimating whether a search query will return high-quality results before the user sees them. However, as the AI landscape shifts from searching for existing images to generating new ones via models like Stable Diffusion, a new challenge has emerged: Prompt Performance Prediction (PPP).

Currently, generative models may produce "hallucinated" or low-quality images when faced with complex or ambiguous prompts, often without any system warning. The PQPP benchmark addresses this gap by providing the first joint framework to evaluate how well we can predict performance across both image generation and image retrieval tasks.

The PQPP Pipeline

The researchers developed a robust pipeline to evaluate 10,200 text prompts, sourcing 10,000 diverse captions from MS COCO (selected via k-means clustering) and 200 challenging prompts from DrawBench.

The PQPP evaluation pipeline for prompt and query performance prediction.

As illustrated in Figure 1, the framework evaluates prompts across two distinct scenarios:

  1. Image Generation: Using state-of-the-art diffusion models, SDXL and GLIDE, to generate images for every prompt.
  2. Image Retrieval: Using vision-language models, CLIP and BLIP-2, to find relevant images within the MS COCO dataset.

To establish a ground truth for performance, the authors conducted a massive human annotation effort, collecting over 1.6 million relevance judgments. This data was then used to train and evaluate various "predictors"—algorithms designed to guess the performance score either before (pre-generation) or after (post-generation) the image is produced.

Key Insights: The Orthogonality of Tasks

The most striking finding of the study is the very low correlation between generation and retrieval performance. The Pearson correlation between human-rated generation performance and retrieval precision was only 0.135.

This suggests that the "difficulty" of a prompt is task-dependent. For instance, a prompt like "a man holding a tennis racquet while bending his legs" is relatively easy for retrieval because image databases are rich in sports photography. However, it remains difficult for generative models, which often struggle with specific, complex human poses. This "orthogonality" proves that specialized tools are required to help users navigate these two fundamentally different AI tasks.

Performance Prediction Results

The study tested several predictor architectures to determine which could best anticipate model success:

  • BERT as a Strong Baseline: Surprisingly, a fine-tuned BERT model—which only analyzes the text—performed nearly as well as "post-generation" models that analyze the resulting images. This suggests that much of a prompt's difficulty is inherent in its linguistic structure.
  • Top Predictors: For the generation task, a fine-tuned CLIP model achieved the highest correlation with human judgments (Pearson 0.649).
  • Generalization: Predictors trained on one model (e.g., SDXL) maintained statistically significant predictive power when tested on another (e.g., GLIDE), confirming the benchmark's utility for cross-model evaluation.

Conclusion and Future Work

PQPP establishes a vital foundation for understanding prompt difficulty in the era of generative AI. By identifying the divergence between generation and retrieval, this work paves the way for more intelligent user interfaces. Future research will focus on developing hybrid predictors that combine text-only and image-aware features, as well as exploring "automatic reformulation," where systems use these predictions to rewrite difficult prompts into easier ones on behalf of the user.

Summary updated

BibTeX
@inproceedings{poesina2025pqpp,
  author = {Eduard Poesina and Adriana Valentina Costache and Adrian-Gabriel Chifu and Josiane Mothe and Radu Tudor Ionescu},
  title = {{PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02668},
}
Figure