Feature Proliferation - the "Cancer" in StyleGAN and its Treatments

Shuang Song, Yuanbang Liang, Jing Wu, Yu-Kun Lai, Yipeng Qin

Abstract

StyleGAN models often produce visual artifacts due to a phenomenon termed "Feature Proliferation," where high-magnitude features dominate and multiply across layers. While the standard "truncation trick" mitigates these artifacts by pulling latent codes toward the mean, it significantly reduces image diversity. This paper introduces a surgical feature rescaling method that identifies and modulates risky features in early layers. This approach effectively "cures" artifacts while preserving fine details and diversity, outperforming traditional truncation across multiple benchmarks.

Introduction

StyleGAN is a titan in the world of image synthesis, but it isn't perfect. It often produces "artifacts"—strange, non-realistic blobs or distortions in generated images. To fix this, researchers typically use the truncation trick, a post-processing step that pulls the latent code toward the average.

While effective at improving quality, the truncation trick is a blunt instrument. It works in a high-level latent space, meaning it often "throws the baby out with the bathwater" by sacrificing image diversity and erasing unique details (like specific facial hair or accessories) just to get rid of the glitches. The authors sought to find a more "fine-grained" way to fix these artifacts without losing the variety that makes GANs useful.

Methods: Identifying and Treating the "Cancer"

The authors' approach moves the intervention from the high-level latent space down into the feature space of the generator. They identify a process called Feature Domination, where a small number of features with abnormally large values dominate the weighted sum of a neuron. When these dominant features reproduce and increase in ratio through subsequent layers, it is termed Feature Proliferation.

The Statistical Heuristic

To find these "risky" features, the researchers look for feature maps that deviate significantly from the mean of their distribution across the training set. They use a specific formula ($r_{l,j}$) to calculate the "risk" of a feature based on its mean and standard deviation.

Statistical distribution of input feature maps used to identify risky features.

Surgical Feature Rescaling

Instead of normalizing the entire output, the method only rescales the identified risky features. By dividing these features by a factor proportional to their risk and a scaling hyperparameter ($p$), they dampen the "cancerous" growth before it can ruin the final image. This method follows a "no retraining" philosophy, meaning it can be applied to existing, pre-trained StyleGAN models without the massive computational cost of training from scratch.

Key Results

The proposed method was evaluated using standard high-resolution datasets including FFHQ (faces), MetFace (portraits), and AFHQ-Cat.

Detail Preservation and Diversity

Qualitatively, the method shines where the truncation trick fails. While truncation often removes eyeglasses or beards to fix artifacts, the rescaling method successfully removes the artifacts while keeping fine facial details intact. Quantitative analysis using Precision and Recall showed that the proposed method achieves high image quality with much less sacrifice to diversity (recall) compared to the truncation trick.

Comparison between images with artifacts (top) and the "cured" results using surgical feature rescaling (bottom).

Quantitative Superiority

The method consistently outperformed the truncation trick across multiple metrics, including FID (Fréchet Inception Distance), PSNR, and SSIM. For example, on StyleGAN2 MetFace, the FID improved from 23.02 (truncation) to 17.51 (ours). Furthermore, the method remains fully compatible with existing StyleGAN operations like latent space interpolation and image editing.

Conclusion and Future Work

The authors traced the root cause of Feature Proliferation to a statistical flaw: StyleGAN models assume input activations are independent and identically distributed (i.i.d.) with unit variance, which is not true in practice. By intervening surgically in the feature space, they target the specific channels causing the problem while leaving the rest of the image's "DNA" intact.

While highly effective, the heuristic might still occasionally dampen useful features or fail to catch extremely subtle artifacts. Future work will investigate if this "cancerous" proliferation is a universal problem in other deep learning architectures beyond StyleGAN.

Summary updated

Abstract

Despite the success of StyleGAN in image synthesis, the images it synthesizes are not always perfect and the well-known truncation trick has become a standard post-processing technique for StyleGAN to synthesize high quality images. Although effective, it has long been noted that the truncation trick tends to reduce the diversity of synthesized images and unnecessarily sacrifices many distinct image features. To address this issue, in this paper, we first delve into the StyleGAN image synthesis mechanism and discover an important phenomenon, namely Feature Proliferation, which demonstrates how specific features reproduce with forward propagation. Then, we show how the occurrence of Feature Proliferation results in StyleGAN image artifacts. As an analogy, we refer to it as the "cancer" in StyleGAN from its proliferating and malignant nature. Finally, we propose a novel feature rescaling method that identifies and modulates risky features to mitigate feature proliferation. Thanks to our discovery of Feature Proliferation, the proposed feature rescaling method is less destructive and retains more useful image features than the truncation trick, as it is more fine-grained and works in a lower-level feature space rather than a high-level latent space. Experimental results justify the validity of our claims and the effectiveness of the proposed feature rescaling method. Our code is available at https://github.com/songc42/Feature-proliferation.
BibTeX
@inproceedings{song2023feature,
  author = {Shuang Song and Yuanbang Liang and Jing Wu and Yu-Kun Lai and Yipeng Qin},
  title = {{Feature Proliferation - the "Cancer" in StyleGAN and its Treatments}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00224},
}
Figure