A parametric texture model based on deep convolutional features closely matches texture appearance for humans

Thomas S. A. Wallis, Christina M. Funke, Alexander S. Ecker, Leon A. Gatys, Felix A. Wichmann, Matthias Bethge

TL;DR

This paper investigates how well deep convolutional neural networks (CNNs), specifically those trained for object recognition, can model human perception of texture. They found that while CNN features are powerful, no single model perfectly captures texture appearance across all conditions, suggesting a nuanced interplay between model complexity, viewing conditions, and texture type.

What Problem Does This Paper Address?

Understanding how humans perceive texture is a fundamental challenge in computer vision and neuroscience. Textures, the "stuff" of our visual world (like bark, cloth, or gravel), are distinct from "things" (objects). Humans are remarkably good at distinguishing subtle texture variations, which is crucial for tasks like material identification and scene segmentation. While traditional models like the Portilla-Simoncelli (PS) model have been influential, they rely on hand-crafted statistics. The advent of deep CNNs, which learn features automatically from data, offers a new avenue for modeling texture perception. This paper aims to bridge the gap between these powerful CNN features and human perception by psychophysically evaluating how well CNN-generated textures match human judgments, and comparing them to established models.

Methods

The authors employed a psychophysical approach using a three-alternative oddity task. In this task, observers were presented with three texture patches and had to identify the one that was "different" – either a model-generated synthesis or an original texture. This design ensures that observers must discriminate based on subtle appearance differences rather than simply picking out an "unnatural" looking image.

They compared three main texture models:

  1. Portilla and Simoncelli (PS) model: A classic parametric model based on joint statistics of complex wavelet coefficients.
  2. CNN texture model (Gatys et al., 2015): This model uses features extracted from intermediate layers of a pre-trained VGG-19 CNN. Texture synthesis is achieved by matching Gram matrices of these features.
  3. CNN + power spectrum model: An extension of the CNN model that additionally matches the power spectrum of the original image to better capture long-range correlations.

Two viewing conditions were used:

  • Parafoveal: Test patches were briefly presented to the near-periphery, mimicking conditions where detailed inspection is not possible.
  • Inspection: Observers could make eye movements and freely inspect all three patches, allowing for foveal viewing.

Datasets

The study used twelve unique texture images sourced from www.textures.com (used with permission). These images were selected to represent a variety of texture types, including those with periodicity, asymmetry, and size gradients. For each original image, ten synthesized texture images were generated by each of the tested models.

Key Insights and Ideas

  • Deep CNNs capture texture appearance, but not perfectly: The core finding is that features from deep CNNs, particularly those trained for object recognition, are surprisingly effective at generating textures that humans find indistinguishable from real ones.
  • Viewing conditions matter: The effectiveness of different models varied significantly between parafoveal (brief, peripheral) and inspection (foveal) viewing conditions.
  • No single "best" model: The study highlights that there isn't a universally superior model. The "best" model depends on the specific texture, the viewing conditions, and the desired level of perceptual fidelity.
  • Power spectrum constraint helps for periodic textures: Adding a power spectrum constraint to the CNN model significantly improved its ability to match textures with strong periodic regularities (like tiles or metal grating).
  • Human perception is complex: Even advanced models struggle with certain textures (e.g., "Candy"), suggesting that human texture perception involves more than just the features captured by current CNN architectures.

Key Results

  • Parafoveal viewing: In brief, peripheral viewing, observers were unable to discriminate 10 out of 12 original images from CNN model images. Remarkably, the simpler PS model performed slightly better than the CNN model in this condition.
  • Foveal inspection: Under free viewing, CNN features captured appearance substantially better than the PS model (9 vs. 4 textures).
  • Power spectrum enhancement: The CNN + power spectrum model improved appearance matching for two textures that the basic CNN model failed to capture.
  • Failure cases: None of the tested models could produce indiscriminable images for one specific texture ("Candy") under inspection conditions.
  • PS model's strength: The PS model was effective for parafoveal viewing, suggesting its statistics are relevant for peripheral texture processing.

Limitations and Future Work

  • Limited number of textures: The study used only 12 texture images, which may not fully represent the diversity of natural textures. The authors suggest that future work could identify more texture "clusters" and use representative images from each cluster for more generalizable testing.
  • Image size and feature scaling: The study used a fixed input image size (256x256 pixels) and CNN feature scaling. The authors acknowledge that the relationship between texture element size and model performance might be complex and warrants further investigation.
  • Gradient descent iterations: The number of gradient descent iterations used for synthesis could influence fidelity. While they found that most of the structural information was captured early in the process, more iterations might offer marginal improvements.
  • "Candy" image: The failure to match the "Candy" image, which features large textons, glossiness, and occlusions, highlights limitations in current CNN feature representations and suggests it as a benchmark for future models.
  • Parsimonious models: A key challenge is to compress the rich feature representations learned by CNNs into a minimal, parsimonious set of statistics that still capture essential aspects of texture appearance.
  • Categorical vs. Perceptual Losslessness: The study focused on "categorical losslessness" (syntheses are perceived as the same category as originals). Future work could explore "perceptual losslessness" (syntheses are indistinguishable from originals), which is more relevant for understanding information loss in visual processing.

Summary updated

Abstract

Our visual environment is full of texture-"stuff" like cloth, bark, or gravel as distinct from "things" like dresses, trees, or paths-and humans are adept at perceiving subtle variations in material properties. To investigate image features important for texture perception, we psychophysically compare a recent parametric model of texture appearance (convolutional neural network [CNN] model) that uses the features encoded by a deep CNN (VGG-19) with two other models: the venerable Portilla and Simoncelli model and an extension of the CNN model in which the power spectrum is additionally matched. Observers discriminated model-generated textures from original natural textures in a spatial three-alternative oddity paradigm under two viewing conditions: when test patches were briefly presented to the near-periphery ("parafoveal") and when observers were able to make eye movements to all three patches ("inspection"). Under parafoveal viewing, observers were unable to discriminate 10 of 12 original images from CNN model images, and remarkably, the simpler Portilla and Simoncelli model performed slightly better than the CNN model (11 textures). Under foveal inspection, matching CNN features captured appearance substantially better than the Portilla and Simoncelli model (nine compared to four textures), and including the power spectrum improved appearance matching for two of the three remaining textures. None of the models we test here could produce indiscriminable images for one of the 12 textures under the inspection condition. While deep CNN (VGG-19) features can often be used to synthesize textures that humans cannot discriminate from natural textures, there is currently no uniformly best model for all textures and viewing conditions.
BibTeX
@article{wallis2017parametric,
  author = {Thomas S. A. Wallis and Christina M. Funke and Alexander S. Ecker and Leon A. Gatys and Felix A. Wichmann and Matthias Bethge},
  title = {{A parametric texture model based on deep convolutional features closely matches texture appearance for humans}},
  journal = {Journal of Vision},
  year = {2017},
  doi = {10.1167/17.12.5},
}

Loading related content...

Figure