Primal sketch: Integrating structure and texture

Cheng-en Guo, Song Chun Zhu, Ying Nian WuCVIU 2007 · DOI

TL;DR

This paper introduces the "primal sketch" model, a novel generative framework that unifies the representation of geometric structures (edges, ridges) and stochastic textures within a single image model. By dividing an image into structure and texture domains and employing sparse coding for structures and Markov random fields for textures, the model achieves a seamless integration of these visual elements, paving the way for more comprehensive image understanding and synthesis.

What Problem Does This Paper Address?

Natural images are composed of both distinct geometric structures (like edges and corners) and complex, repetitive textures. Traditionally, these have been modeled separately, leading to fragmented representations and difficulties in tasks requiring a holistic understanding of an image. For instance, objects that appear as structures at one viewing distance might look like textures at another. This paper addresses the challenge of creating a unified representational and computational framework that can effectively model and integrate both geometric structures and stochastic textures within a single generative image model.

Methods

The core of the proposed "primal sketch" model lies in its hybrid approach:

  • Image Domain Division: The image lattice is explicitly divided into a "structure domain" and a "texture domain."
  • Structure Representation (Sparse Coding): Geometric structures (edges, ridges, corners, junctions) are modeled using a sparse coding approach. This involves representing these structures with a small number of "coding functions" that have explicit geometric and photometric parameters. These functions are inspired by models of simple cells in the visual cortex, such as Gabor filters or Difference of Gaussian filters. The collection of these structures forms a "sketch graph."
  • Texture Representation (Markov Random Field): The texture domain is modeled using a Markov random field (MRF). Texture patterns are characterized by statistical summaries of filter responses, specifically marginal histograms. The MRF then "in-paints" the texture domain, guided by these statistics and the boundaries defined by the structure domain.
  • Sketch Pursuit Algorithm: A novel algorithm called "sketch pursuit" is introduced for fitting the model to an image. It's a multi-phase process that first detects edge and ridge candidates, then sequentially adds coding functions to build the sketch graph, and finally refines the graph by fixing corners and junctions using reversible graph operators.
  • Texture Clustering and Synthesis: After the structure domain is defined, the texture domain is segmented into homogeneous regions using k-means clustering based on local filter response histograms. Textures are then synthesized within these regions.

Datasets

The paper uses "real images" for experiments. Specific datasets are not named, but examples include a horse-riding scene, a building facade, and images of a zebra and a fish. The focus is on demonstrating the model's capability on natural images rather than on a specific benchmark dataset.

Key Insights and Ideas

  • Unified Representation: The most significant insight is the successful integration of structure and texture into a single generative model. This moves beyond separate treatments of these visual components.
  • Texture as "Recycled" Filter Responses: The paper proposes that texture statistics arise from the recycling of filter responses that fail to detect clear edges or ridges. This provides a novel perspective on the relationship between structure and texture.
  • Structure-Guided Texture Synthesis: Textures are synthesized with the structure domain acting as boundary conditions, ensuring a seamless integration between the two components.
  • Sketch Graph as a Rich Structural Representation: The sketch graph, composed of parameterized edge and ridge segments, captures not just the presence but also the geometric and photometric properties of structures, going beyond simple edge maps.
  • Biological Plausibility: The use of filters inspired by V1 neurons and the hierarchical processing (linear filtering, then combining responses) aligns the model with biological vision systems.

Key Results

  • The primal sketch model successfully captures both structural and textural aspects of real images, as demonstrated by synthesized images that are visually similar to the originals (Figs. 13, 15, 16, 17).
  • The model provides a lossy image coding scheme. For the example in Fig. 1, the synthesized image was coded in 3049 bytes (0.04 byte/pixel), significantly less than lossless JPEG 2000.
  • The sketch pursuit algorithm effectively extracts structural information, building a detailed sketch graph that includes corners and junctions.
  • The texture synthesis component, guided by the structure boundaries, produces visually coherent textures.

Limitations and Future Work

  • Structure-Texture Relationship: The paper acknowledges that the relationship between the sketch graph and the segmentation of the texture part is not explicitly modeled.
  • Lighting and Shading: The current model does not account for structures caused by lighting and shading effects.
  • Piecewise Homogeneous Textures: The model assumes piecewise homogeneous textures and cannot handle textures on slanted surfaces.
  • Future Work: The authors suggest investigating more sophisticated models for texture segmentation and exploring how to incorporate lighting and shading. They also note that the computational efficiency and theoretical guarantees of the algorithm require further investigation.

Summary updated

BibTeX
@article{guo2007primal,
  author = {Cheng-en Guo and Song Chun Zhu and Ying Nian Wu},
  title = {{Primal sketch: Integrating structure and texture}},
  journal = {Comput. Vis. Image Underst.},
  year = {2007},
  doi = {10.1016/j.cviu.2005.09.004},
}
Figure