GANalyze: Toward Visual Definitions of Cognitive Image Properties.

Lore Goetschalckx, Alex Andonian, Aude Oliva, Phillip Isola

Abstract

GANalyze is a framework that leverages Generative Adversarial Networks (GANs) to provide visual definitions for abstract cognitive properties such as memorability, aesthetics, and emotional valence. By navigating the latent space of a pretrained GAN, the system generates image "clones" that vary along a specific cognitive dimension while maintaining semantic category. This approach moves beyond black-box prediction toward visual explanation, revealing the specific features—such as object size and colorfulness—that drive human cognitive responses.

Introduction

While deep learning models can accurately predict if an image is memorable or aesthetically pleasing, they often fail to explain why. Abstract attributes like "memorability" lack a single, concrete visual definition. Previous attempts to understand these properties relied on sorting large datasets of real images, which makes it difficult to isolate specific causal factors because different images vary in countless ways. GANalyze addresses this by generating synthetic image manifolds where only the cognitive property of interest is manipulated.

Methodology

The GANalyze framework consists of three primary components:

  1. The Generator ($G$): A pretrained BigGAN that produces high-quality images from a latent vector $z$ and a class vector $y$.
  2. The Assessor ($A$): A pretrained model (e.g., MemNet for memorability or AestheticsNet for beauty) that scores images for a specific property.
  3. The Transformer ($T$): A novel component that learns a specific direction $\theta$ in the GAN's latent space. By moving along this direction by an amount $\alpha$ (the "metaphorical knob"), the Transformer modifies the original $z$ vector to produce a new image that the Assessor will score higher or lower.
Figure 2: Transformation of a dog image across increasing cognitive scores, demonstrating changes in composition and subject prominence.

Key Insights and Visual Definitions

The framework reveals "emerging factors" that define cognitive properties. For memorability, the model discovered that increasing object size, centering the subject, circularity, and simplification (reducing background clutter) all lead to higher scores. Crucially, behavioral experiments involving thousands of participants on Amazon Mechanical Turk confirmed that these GAN-generated changes have a causal effect on human memory performance, proving the model isn't just exploiting adversarial noise.

Figure 4: Latent space navigation modifying a pepper's visual attributes to increase its predicted cognitive score.

Furthermore, the model demonstrates property independence. To make an image more aesthetic, it introduces bokeh (background blur), adjusts lighting, and shifts the color palette. To increase emotional valence, it might make a teddy bear "smile" or make a landscape's sky bluer. All generated images maintained high realism, with Frechet Inception Distance (FID) scores below 40.

Results and Limitations

Quantitative analysis shows the Transformer successfully shifts Assessor scores with a significant slope ($\beta = 0.68$). In human trials, "hit rates" for correctly identifying repeated images increased significantly as the transformation parameter $\alpha$ increased ($\beta = 1.92$ in log odds).

Despite its success, GANalyze is currently limited to images generated by the GAN itself due to the "inversion problem" of mapping real photos back into latent space. Additionally, the model inherits biases from its training data (ImageNet). Future work aims to overcome these hurdles, potentially enabling applications in graphic design, advertising, and education by allowing users to edit real-world photographs to enhance specific cognitive impacts.

Summary updated

Abstract

We introduce a framework that uses Generative Adversarial Networks (GANs) to study cognitive properties like memorability. These attributes are of interest because we do not have a concrete visual definition of what they entail. What does it look like for a dog to be more memorable? GANs allow us to generate a manifold of natural-looking images with fine-grained differences in their visual attributes. By navigating this manifold in directions that increase memorability, we can visualize what it looks like for a particular generated image to become more memorable. The resulting ``visual definitions" surface image properties (like ``object size") that may underlie memorability. Through behavioral experiments, we verify that our method indeed discovers image manipulations that causally affect human memory performance. We further demonstrate that the same framework can be used to analyze image aesthetics and emotional valence. ganalyze.csail.mit.edu.
BibTeX
@inproceedings{goetschalckx2019ganalyze,
  author = {Lore Goetschalckx and Alex Andonian and Aude Oliva and Phillip Isola},
  title = {{GANalyze: Toward Visual Definitions of Cognitive Image Properties.}},
  booktitle = {ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=oA4L43Oj5tR},
}
Figure