Image Modeling: From Handcrafted Priors to Foundation Image Models

A. Murat TekalparXiv 2026 · arXiv

TL;DR

This paper provides a comprehensive unification of six decades of image modeling, tracing the evolution from handcrafted mathematical priors to modern, data-driven foundation models. By grounding both classical inverse problem solving and contemporary image generation in a single Bayesian framework—specifically the Stein Score Function—the author demonstrates that the pursuit of the "denoising score" is the common thread linking signal processing and modern AI.

What Problem Does This Paper Address?

For decades, the signal processing and machine learning communities operated on seemingly parallel tracks. Signal processing focused on inverse problems (restoring images from noise, blur, or downsampling) using "handcrafted" mathematical rules to describe what a natural image looks like. Meanwhile, machine learning pivoted toward generative modeling (creating new images from scratch) using massive datasets.

The core challenge is defining the "natural image manifold"—the incredibly thin slice of high-dimensional space where real-looking images exist. The paper addresses the gap between these two worlds, showing how the mathematical tools used to restore a blurry photo in the 1990s are fundamentally the same as those used by DALL-E or Stable Diffusion today.

Methods

The paper organizes the evolution of image modeling into several distinct eras, unified by the Bayesian prior $p(x)$:

  1. Handcrafted Priors (1970s–2000s): Methods like Tikhonov regularization ($L_2$ smoothness) and Total Variation ($L_1$ sparsity) used explicit formulas to penalize "unnatural" features like noise.
  2. The Algorithmic Bridge (2013): The "Denoiser as a Prior" paradigm (e.g., Plug-and-Play, RED) showed that we don't need an explicit formula for an image prior. Instead, any high-quality denoiser can act as an implicit prior within an optimization loop.
  3. Trajectory-Based Generative Models (2020–Present): This includes Denoising Diffusion Probabilistic Models (DDPM) and Score-Based Models. These treat image generation as a process of systematically removing noise along a trajectory.
  4. Foundation Image Models (2023–2026): The current state-of-the-art, which replaces traditional convolutional U-Nets with Diffusion Transformers (DiT). These models leverage "Scaling Laws"—the observation that increasing model size and data leads to emergent zero-shot capabilities.
The timeline of evolution of image modeling from hand-crafted priors to foundation image models.

Datasets

As a review and unification paper, it does not introduce a new dataset. However, it highlights the shift in data usage:

  • Early Era: Small, specialized datasets (e.g., human faces).
  • Foundation Era: Web-scale, multimodal datasets spanning billions of image-text pairs (e.g., those used to train CLIP, Stable Diffusion, and FLUX.1).

Key Insights and Ideas

  • The Stein Score Function is King: The gradient of the log-prior, $\nabla_x \log p(x)$, is the "universal engine." Whether you are performing MAP estimation (optimization) or posterior sampling (generation), you are essentially navigating the score field of the image manifold.
  • Tweedie’s Formula: This classical statistical result provides the rigorous link between signal processing and AI. It proves that an optimal MMSE denoiser is, by definition, a score estimator for the data distribution.
  • From U-Net to DiT: The architectural shift from U-Nets (which have a strong inductive bias for local pixels) to Transformers (which use global self-attention) is what allowed image models to finally benefit from the same scaling laws that powered Large Language Models (LLMs).
  • Perception-Distortion Tradeoff: The paper emphasizes that you cannot have it all. Minimizing distortion (like MSE) often leads to blurry results, while maximizing perceptual realism (making it look "real") can lead to "hallucinations" where the model invents details not present in the original data.

Key Results

The paper synthesizes the quantitative progress of the field:

  • Efficiency Gains: Transitioning from stochastic SDE paths to deterministic "Rectified Flows" has reduced the number of sampling steps required for high-fidelity images from 100+ down to as few as 4 to 10 steps.
  • Zero-Shot Versatility: Foundation models (FIMs) now act as "universal priors." A single frozen model can be plugged into different iterative solvers to solve diverse inverse problems (super-resolution, inpainting, deblurring) without any task-specific retraining.
  • Scaling: The transition to DiT architectures has enabled models to scale parameters into the billions, leading to models like Sora and Midjourney v6 that appear to have "learned" physical properties like gravity and light reflection simply by observing data.

Limitations and Future Work

  • Generative Hallucinations: While foundation models produce stunningly realistic images, they still struggle with accuracy in scientific or medical contexts where "inventing" detail is unacceptable.
  • Fine-Grained Control: Achieving precise, pixel-level control over generated content remains an active area of research (e.g., through techniques like ControlNet).
  • Evaluation Metrics: The field is moving away from PSNR/SSIM toward perceptual metrics like FID and LPIPS, but the author notes that the "best" metric is still highly task-dependent.

Summary updated

Abstract

This paper unifies six decades of image modeling, evolving from handcrafted mathematical tractable priors to solve inverse problems in imaging to data-driven deep foundation image models for both image generation and solving inverse problems, under a common Bayesian framework that formulates image modeling as an implicit or explicit probabilistic characterization of a data manifold. Every image restoration or generative task fundamentally seeks to answer the same question: What makes an image look like a real image? In the Bayesian framework, this question is encapsulated by the prior probability density p(x), or its more optimization-friendly counterpart, the log-prior log p(x). While early works in solving inverse problems, such as Tikhonov, Total Variation (TV), and Non-local priors relied on explicit handcrafted definitions of log prior to model local smoothness, modern techniques, such as Plug-and-Play (PnP) priors and Denoising Diffusion models, are rooted in algorithmic uncoupling and implicit manifold learning that apply to both inverse problems and conditional image generation. By grounding imaging inverse problems and conditional image generation under a single mathematical engine: the gradient of the log-prior, or the Stein Score Function, we bridge the gap between signal processing and machine learning communities, turning what seems like parallel developments into a single, evolution of image modeling from handcrafted analytical models to models learned from big data, demonstrating that the pursuit of the denoising score is the common thread connecting variational optimization (in MAP estimation) to posterior sampling (in modern foundation image models).
BibTeX
@misc{tekalp2026image,
  author = {A. Murat Tekalp},
  title = {{Image Modeling: From Handcrafted Priors to Foundation Image Models}},
  year = {2026},
  eprint = {2610.36646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36646},
}
Figure