Image analogies

Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, David H. Salesin

TL;DR

Hertzmann et al. introduce Image Analogies, a framework that learns to apply complex image filters by example. By providing a single pair of "unfiltered" and "filtered" source images, the algorithm can synthesize a new version of a target image that mimics the learned transformation, enabling everything from artistic stylization to super-resolution without explicit programming.

What Problem Does This Paper Address?

In traditional computer graphics, creating a specific visual effect—like a watercolor filter or an embossing tool—requires a programmer to manually define the underlying mathematical transformation. This is both labor-intensive and limited by the programmer's ability to formalize aesthetic styles.

The authors tackle the challenge of example-based image processing. Instead of writing code for a specific filter, can we simply show a computer a "before" and "after" pair and ask it to "do that to this new image"? The goal is to solve the analogy $A : A' :: B : B'$, where $A$ and $A'$ are the source pair, $B$ is a new target, and $B'$ is the synthesized result.

Methods

The core of the approach is a multiscale autoregression algorithm that builds upon earlier work in texture synthesis.

  1. Multiscale Representation: The algorithm constructs Gaussian pyramids for all images ($A, A', B$). Synthesis of the output $B'$ proceeds from the coarsest to the finest resolution.
  2. Neighborhood Matching: To determine the color of a pixel $q$ in the output $B'$, the algorithm looks at the local neighborhood of $q$ in both $B$ and the partially synthesized $B'$. It then searches the source pair $(A, A')$ for a pixel $p$ whose neighborhoods in $A$ and $A'$ most closely match those of $q$.
  3. Two-Pronged Search Strategy:
    • Approximate Search: Uses an Approximate Nearest Neighbor (ANN) search to find the best statistical match in feature space.
  4. Coherence Search: Inspired by Ashikhmin’s texture synthesis, this search favors pixels that are spatially contiguous with their neighbors in the source image. This helps maintain the "flow" of textures and brush strokes.
  5. A coherence parameter ($\kappa$) allows users to trade off between raw pixel accuracy and visual continuity.
  6. Feature Vectors: Instead of just using raw RGB values, the algorithm often operates in luminance space (the Y-channel of YIQ) to avoid the "curse of dimensionality" and better match human visual perception. For line art, it can also incorporate steerable filter responses to capture edge orientations.
The neighborhood matching process across multiple scales.

Datasets

The paper does not rely on modern large-scale datasets. Instead, it demonstrates results using specific training pairs:

  • Artistic Works: Details from paintings by Van Gogh (Starry Night) and Manet to learn painterly styles.
  • Textures: Samples from the MIT VisTex database for texture synthesis and transfer.
  • Photographs: Landscape and portrait photography used for blurring, embossing, and super-resolution tests.
  • Hand-painted Maps: Simple color-coded "label" images used for the "texture-by-numbers" application.

Key Insights and Ideas

  • Generality is Key: The most striking insight is that a single, relatively simple neighborhood-matching algorithm can handle vastly different tasks—blurring, artistic stylization, and texture synthesis—just by changing the input pair.
  • Coherence Matters: Purely statistical matching often results in "salt and pepper" noise. By forcing the algorithm to copy small patches (coherence), the results look much more natural and "hand-drawn."
  • Luminance Remapping: To apply a style from a dark source image to a bright target image, the authors use a linear map to match the means and variances of the luminance distributions, ensuring the "histograms" of the images are in the same ballpark before matching.
  • Texture-by-Numbers: By using a simplified color map as the "unfiltered" source $A$, users can "paint" complex scenes. For example, painting a blue blob in $B$ can be mapped to "water texture" from $A'$ if the corresponding area in $A$ was also blue.

Key Results

The paper demonstrates impressive qualitative results across several domains:

  • Artistic Filters: Successfully transformed photographs into oil paintings and pen-and-ink drawings that capture the local texture of the original artist's work.
  • Super-resolution: The algorithm can "hallucinate" high-frequency details in a blurred image by learning from a small high-res/low-res training patch.
Example of super-resolution: recovering detail from a blurred target using a small training pair.
  • Texture Transfer: Applying the texture of one object (e.g., a rug) to the shape of another (e.g., a face).
  • Performance: While slow by today's standards (taking minutes to hours on 2001-era hardware), the algorithm was efficient enough to support a basic interactive "painting" mode by only updating local regions.

Limitations and Future Work

  • Low-Level Statistics: The algorithm only "sees" small neighborhoods. It cannot capture high-level semantic structures, such as long, sweeping brush strokes that follow the global contour of a face.
  • Registration Requirements: The source pair $(A, A')$ must be perfectly registered (pixel-aligned). If $A$ is a photo and $A'$ is a hand-painted version, any misalignment will degrade the learned filter.
  • Computational Cost: The search process is expensive. The authors suggest future work in better regression techniques and hardware acceleration.
  • Beyond 2D: The authors speculate that these analogic reasoning principles could eventually be applied to 3D geometry, animation, and even non-visual domains like music synthesis.

Summary updated

Abstract

This paper describes a new framework for processing images by example, called “image analogies.” The framework involves two stages: a design phase, in which a pair of images, with one image purported to be a “filtered” version of the other, is presented as “training data”; and an application phase, in which the learned filter is applied to some new target image in order to create an “analogous” filtered result. Image analogies are based on a simple multi-scale autoregression, inspired primarily by recent results in texture synthesis. By choosing different types of source image pairs as input, the framework supports a wide variety of “image filter” effects, including traditional image filters, such as blurring or embossing; improved texture synthesis, in which some textures are synthesized with higher quality than by previous approaches; super-resolution, in which a higher-resolution image is inferred from a low-resolution source; texture transfer, in which images are “texturized” with some arbitrary source texture; artistic filters, in which various drawing and painting styles are synthesized based on scanned real-world examples; and texture-by-numbers, in which realistic scenes, composed of a variety of textures, are created using a simple painting interface.
BibTeX
@inproceedings{hertzmann2001image,
  author = {Aaron Hertzmann and Charles E. Jacobs and Nuria Oliver and Brian Curless and David H. Salesin},
  title = {{Image analogies}},
  booktitle = {Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques},
  year = {2001},
  doi = {10.1145/383259.383295},
}
Figure