Fantastic Style Channels and Where to Find Them: A Submodular Framework for Discovering Diverse Directions in GANs

Enis Simsar, Umut Kocasari, Ezgi Gülperi Er, Pinar Yanardag

Introduction

Generative Adversarial Networks (GANs), particularly StyleGAN2, have revolutionized image synthesis. Beyond generating high-fidelity images, their latent spaces contain "directions" that correspond to semantic attributes like age, hair color, or lighting. However, discovering these directions typically requires either extensive manual labeling (supervised) or results in highly redundant "knobs" (unsupervised).

The Redundancy Challenge

Current unsupervised methods, such as PCA-based GANSpace, often fail to capture the full breadth of a model's capabilities. In a face model, for instance, an automated search might return dozens of directions that all modify the background while completely missing subtle features like ear shape or clothing details. This redundancy stems from the high dimensionality of StyleGAN2’s StyleSpace ($S$-space), which contains over 9,000 channels. For example, the authors found over 300 channels dedicated to hair and 180 to background/ears; without a diversity constraint, an automated search will likely return multiple redundant ways to change the same attribute.

A Submodular Framework for Discovery

To address this, researchers proposed a novel framework that treats direction discovery as a combinatorial optimization problem using submodular functions. This approach aims to pick a small, representative subset of channels that covers as many unique types of edits as possible without repetition.

1. Quantifying and Clustering Effects

The method first measures the perceptual impact of perturbing individual channels using two metrics: LPIPS (to measure the magnitude of perceptual change) and SSIM (to measure structural similarity). Channels that affect similar image regions (e.g., all channels modifying the nose) are grouped into clusters using Agglomerative Clustering based on their SSIM scores.

2. The Submodular Objective

The core of the framework is a mathematical objective function designed to reward two properties:

  • Coverage: Selecting channels that are highly representative of the overall latent space.
  • Diversity: Applying a "diminishing returns" logic. If a channel from a specific cluster (like "Hair") is already selected, the marginal reward for picking another from the same cluster is significantly reduced (using a log-sum function).
The framework discovers diverse and disentangled directions across various domains, grouping similar manipulations into clusters while ensuring a wide variety of edits.

3. Greedy Optimization

Because the objective function is submodular, a simple "greedy" algorithm—selecting the best remaining channel one by one—is mathematically guaranteed to provide a solution very close to the theoretical optimum.

Key Results and Insights

The framework was evaluated across a wide variety of domains, including FFHQ (faces), Metfaces (portraits), AFHQ Cats, LSUN Cars, and a custom Fashion dataset.

  • Superior Disentanglement: By operating in $S$-space rather than the more common $W$ or $W+$ spaces, the method achieves much cleaner edits. In a study with 25 participants, this method scored 4.32/5.0 for disentanglement, significantly outperforming GANSpace (2.46) and SeFa (2.91).
  • Uncovering Hidden Attributes: Unlike supervised methods, this framework discovered attributes that weren't even in the training labels, such as specific "clothing patterns" or "neck types" in fashion models.
  • Efficiency: The automated nature of the submodular selection removes the need for human-in-the-loop exploration of thousands of channels, effectively avoiding the "redundancy trap."

Conclusion and Future Work

By framing latent direction discovery through the lens of submodular optimization, this work provides a powerful tool for exploring and utilizing pre-trained GANs. While currently optimized for StyleGAN2, the authors suggest the framework could be extended to other architectures and real-image editing via GAN inversion. It successfully navigates the trade-off between manual supervision and redundant unsupervised discovery, offering a diverse and disentangled set of controls for high-quality image manipulation.

Summary updated

Abstract

The discovery of interpretable directions in the latent spaces of pre-trained GAN models has recently become a popular topic. In particular, StyleGAN2 has enabled various image generation and manipulation tasks due to its rich and disentangled latent spaces. However, the discovery of such directions is typically made either in a supervised manner, which requires annotated data for each desired manipulation, or in an unsupervised manner, which requires a manual effort to identify the directions. As a result, existing work typically finds only a handful of directions in which controllable edits can be made. In this study, we design a novel submodular framework that finds the most representative and diverse subset of directions in the latent space of StyleGAN2. Our approach takes advantage of the latent space of channel-wise style parameters, so-called stylespace, in which we cluster channels that perform similar manipulations into groups. Our framework promotes diversity by using the notion of clusters and can be efficiently solved with a greedy optimization scheme. We evaluate our framework with qualitative and quantitative experiments and show that our method finds more diverse and disentangled directions.
BibTeX
@inproceedings{simsar2023fantastic,
  author = {Enis Simsar and Umut Kocasari and Ezgi Gülperi Er and Pinar Yanardag},
  title = {{Fantastic Style Channels and Where to Find Them: A Submodular Framework for Discovering Diverse Directions in GANs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=m009P8UcWv},
}
Figure