TL;DR
GANalyze is a novel framework that navigates the latent space of Generative Adversarial Networks (GANs) to find specific directions that alter high-level cognitive image properties like memorability, aesthetics, and emotional valence. By generating a continuous spectrum of an image shifting along these dimensions, the method provides concrete "visual definitions" of abstract concepts and demonstrates a direct causal link to human psychological behavior.
What Problem Does This Paper Address?
We intuitively understand how to modify simple image attributes—we can zoom in to increase "object size" or tweak a mouth to create a "smile." However, defining abstract cognitive properties like "memorability" or "aesthetics" is much harder. What exactly makes a picture of a dog memorable?
While deep learning models can accurately predict how memorable or beautiful an image is, they operate as black boxes. They fail to explain the underlying visual features responsible for those scores. The authors tackle this gap by using generative models to visually define and isolate the fine-grained visual attributes that drive high-level cognitive perceptions.
Methods
The GANalyze framework introduces a clever three-part architecture:
- Generator ($G$): A pretrained generative network (such as BigGAN or StyleGAN) that takes a latent noise vector $z$ and a class vector $y$ to output a realistic image.
- Assessor ($A$): A pretrained predictor that scores the image on a cognitive property (e.g., MemNet for memorability).
- Transformer ($T$): A trainable module that learns a specific direction vector ($\theta$) within the GAN's latent space.
Instead of modifying pixels directly, the Transformer shifts the latent vector $z$ along the direction $\theta$ by a variable step size $\alpha$ (acting as a "metaphorical knob"): $$T_\theta(z, \alpha) = z + \alpha\theta$$
The network is trained using a Mean Squared Error (MSE) loss function that penalizes the difference between the Assessor's score of the newly generated image and the target score (the original image's score plus $\alpha$). Because the framework uses a class-conditional GAN, it can alter the cognitive property while keeping the underlying object category completely unchanged.
Datasets
- Training Data: The Transformer was trained using a synthetic dataset of 400K latent $z$ vectors sampled from a truncated standard normal distribution, paired with random $\alpha$ values and random ImageNet classes.
- Test Data: A evaluation set consisting of 7.5K total images spanning 750 randomly selected ImageNet classes.
- Human Evaluation: Behavioral data was collected via Amazon Mechanical Turk (AMT). This included a two-alternative forced choice (2AFC) task to evaluate image realness and a repeat-detection visual memory game to measure actual human memory performance.
Key Insights and Ideas
- Emerging Factors of Memorability: By moving along the discovered latent path, the model automatically uncovers several visual features that drive memorability. To make an image more memorable, the framework tends to enlarge the main object, center the subject, simplify the background (reducing clutter), and shift colors toward brighter, redder hues.
- Distinct Latent Directions: Cognitive properties do not share the same visual definitions. While increasing memorability focuses on object size and simplification, increasing aesthetics alters depth of field and lighting. Meanwhile, increasing emotional valence introduces bokeh, makes skies bluer, and makes subjects appear to smile.
- Causal Link to Human Behavior: The learned modifications do not just fool the Assessor network; they successfully alter human memory performance in behavioral tests, proving that the Assessor captured genuine psychological triggers.
Key Results
- Assessor Control: The framework successfully controlled the target properties. For memorability, the mean MemNet score increased significantly with $\alpha$, achieving a regression slope ($\beta$) of 0.68 ($p < 0.001$).
- Human Memory Performance: In the visual memory game, human hit rates increased significantly as $\alpha$ increased ($\beta = 1.92$ for predicted log odds of recognition, $p < 0.001$). This confirms a causal effect on human memory.
- Realness Preserved: Crucially, manipulating these images did not make them look bizarre or unrealistic. Human performance in discriminating GAN-generated images from real photos remained statistically unchanged across different $\alpha$ conditions ($\beta = -0.08, p = 0.55$).
Limitations and Future Work
- Dependent on Latent Codes: GANalyze can currently only modify images that are natively generated by the GAN (where the latent vector $z$ is known). It cannot easily take an arbitrary, user-uploaded photograph without utilizing an additional, highly accurate GAN encoder network.
- Bounded by the Generator: The framework's visual definitions are strictly limited by what the underlying GAN is capable of producing. For instance, because BigGAN struggles to render realistic human beings, the authors could not fully explore human-centric cognitive properties.
- Potential for Misuse: As generative models become more realistic, the ability to algorithmically optimize images to be highly memorable or emotionally manipulative presents notable ethical concerns and risks of misuse.
Summary updated