Abstract
GANalyze is a framework that leverages Generative Adversarial Networks (GANs) to provide visual definitions for abstract cognitive properties such as memorability, aesthetics, and emotional valence. By navigating the latent space of a pretrained GAN, the system generates image "clones" that vary along a specific cognitive dimension while maintaining semantic category. This approach moves beyond black-box prediction toward visual explanation, revealing the specific features—such as object size and colorfulness—that drive human cognitive responses.
Introduction
While deep learning models can accurately predict if an image is memorable or aesthetically pleasing, they often fail to explain why. Abstract attributes like "memorability" lack a single, concrete visual definition. Previous attempts to understand these properties relied on sorting large datasets of real images, which makes it difficult to isolate specific causal factors because different images vary in countless ways. GANalyze addresses this by generating synthetic image manifolds where only the cognitive property of interest is manipulated.
Methodology
The GANalyze framework consists of three primary components:
- The Generator ($G$): A pretrained BigGAN that produces high-quality images from a latent vector $z$ and a class vector $y$.
- The Assessor ($A$): A pretrained model (e.g., MemNet for memorability or AestheticsNet for beauty) that scores images for a specific property.
- The Transformer ($T$): A novel component that learns a specific direction $\theta$ in the GAN's latent space. By moving along this direction by an amount $\alpha$ (the "metaphorical knob"), the Transformer modifies the original $z$ vector to produce a new image that the Assessor will score higher or lower.
Key Insights and Visual Definitions
The framework reveals "emerging factors" that define cognitive properties. For memorability, the model discovered that increasing object size, centering the subject, circularity, and simplification (reducing background clutter) all lead to higher scores. Crucially, behavioral experiments involving thousands of participants on Amazon Mechanical Turk confirmed that these GAN-generated changes have a causal effect on human memory performance, proving the model isn't just exploiting adversarial noise.
Furthermore, the model demonstrates property independence. To make an image more aesthetic, it introduces bokeh (background blur), adjusts lighting, and shifts the color palette. To increase emotional valence, it might make a teddy bear "smile" or make a landscape's sky bluer. All generated images maintained high realism, with Frechet Inception Distance (FID) scores below 40.
Results and Limitations
Quantitative analysis shows the Transformer successfully shifts Assessor scores with a significant slope ($\beta = 0.68$). In human trials, "hit rates" for correctly identifying repeated images increased significantly as the transformation parameter $\alpha$ increased ($\beta = 1.92$ in log odds).
Despite its success, GANalyze is currently limited to images generated by the GAN itself due to the "inversion problem" of mapping real photos back into latent space. Additionally, the model inherits biases from its training data (ImageNet). Future work aims to overcome these hurdles, potentially enabling applications in graphic design, advertising, and education by allowing users to edit real-world photographs to enhance specific cognitive impacts.
Summary updated