A Generalization Theory for Zero-Shot Prediction

Ronak Mehta, Zaid Harchaoui

In 2021, OpenAI’s CLIP redefined computer vision by achieving high accuracy on ImageNet without using a single ImageNet training label. While practitioners have since rushed to build "foundation models" based on this paradigm, the theoretical community has struggled to explain exactly why this works.

Standard statistical learning theory is built on the idea of minimizing risk using labeled data $(X, Y)$. Zero-Shot Prediction (ZSP) breaks this paradigm by using an intermediate modality—typically text captions, $Z$—as a bridge. Models are pre-trained on $(X, Z)$ pairs and then use prompts to link labels $Y$ to text $Z$ during inference.

This paper provides the first rigorous theoretical framework for understanding ZSP, moving beyond empirical intuition to quantifiable guarantees. By decomposing ZSP error into distinct mathematical components, the authors offer a roadmap for understanding why prompting strategies and massive pre-training datasets are so effective.

The Theoretical Framework

To formalize ZSP, the authors treat it as an "indirect predictor" and compare it to a "direct predictor" (the ideal supervised model trained directly on labels). They identify two primary ways to view the ZSP pipeline:

  1. The Conditional Mean Approach: This views pre-training as learning a mapping (an operator) between modalities, and prompting as an approximation of the class-conditional mean in the text space.
  2. The Information Density Approach: This views pre-training as learning the "likelihood ratio" (Radon-Nikodym derivative) between the joint distribution of images and text versus their marginal distributions.
Causal graph illustrating the relationship between the image ($X$), text caption ($Z$), and label ($Y$).

The core contribution of the paper is a Generalization Bound that decomposes the Mean Squared Error (MSE) of a zero-shot model into three distinct penalties:

  • Residual Dependence: This is the information-theoretic cost of using text as a proxy for labels. If the image ($X$) and label ($Y$) are still highly dependent after accounting for the caption ($Z$), ZSP will always lag behind supervised learning. Theoretically, ZSP is only perfect if $X$ and $Y$ are conditionally independent given $Z$ (i.e., the caption captures everything relevant about the image-label relationship).
  • Prompt Bias: The error introduced when our chosen prompts (e.g., "a photo of a [class]") fail to perfectly match the distribution of text seen during pre-training.
  • Estimation Error: The standard statistical penalty for having a finite number of pre-training examples ($N$) and a finite number of prompts ($M$).

Key Insights: Prompting as Statistical Estimation

We often think of "prompt engineering" as a heuristic hack. However, this framework reveals that it is actually a form of Monte Carlo estimation. Increasing the number of prompts ($M$) reduces the variance of the predictor, much like increasing training samples ($N$) does in supervised learning.

The authors empirically confirmed a "Scaling Law of Prompts": ZSP accuracy scales with the number of prompts, typically saturating between 10 and 60 prompts where adding more text descriptions yields diminishing returns.

Furthermore, the theory explains why using Large Language Models (LLMs) to generate prompts works so well. When an LLM generates diverse, descriptive prompts (e.g., "Describe what a cat looks like" rather than just "a photo of a cat"), it creates a better approximation of the true class-conditional distribution. This directly shrinks the Prompt Bias term in the error decomposition. In empirical tests using LlaMA 3 to generate prompts (the CuPL method), top-5 accuracy for a ResNet50-CLIP model improved by nearly 20% on the DTD (textures) dataset compared to standard template prompts.

Validation and Future Directions

The authors validated their theory using a mix of simulated data and real-world pre-trained models (ResNet50, NLLB-CLIP, ViT-B/32) across downstream benchmarks like ImageNet-1k, DTD, Flowers 102, FGVC Aircraft, and SUN397. Through controlled simulations, they demonstrated that as "Residual Dependence" approaches zero (meaning the text perfectly captures the image-label link), the zero-shot model's performance converges to that of a direct supervised learner.

While this framework provides a crucial foundation, several avenues remain for future work. The current theory focuses heavily on classification; extending these guarantees to more complex tasks like object detection or image generation is a promising next step. Additionally, while the paper tracks the mismatch between pre-training and evaluation distributions, a deeper dive into out-of-distribution generalization and causal generative modeling will be necessary to fully map the fundamental links between images, captions, and labels.

Summary updated

Abstract

A modern paradigm for generalization in machine learning and AI consists of pre-training a task-agnostic foundation model, generally obtained using self-supervised and multimodal contrastive learning. The resulting representations can be used for prediction on a downstream task for which no labeled data is available. We present a theoretical framework to better understand this approach, called zero-shot prediction. We identify the target quantities that zero-shot prediction aims to learn, or learns in passing, and the key conditional independence relationships that enable its generalization ability.
BibTeX
@inproceedings{mehta2025generalization,
  author = {Ronak Mehta and Zaid Harchaoui},
  title = {{A Generalization Theory for Zero-Shot Prediction}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=kJQgMGLrow},
}
Figure