BSD-GAN: Branched Generative Adversarial Network for Scale-Disentangled Representation Learning and Image Synthesis

Zili Yi, Zhiqin Chen, Hao Cai, Wendong Mao, Minglun Gong, Hao ZhangarXiv 2018 · arXiv

TL;DR

BSD-GAN introduces a progressive, multi-branch training strategy that forces specific segments of the input noise vector to learn image features at different scales (from coarse shapes to fine textures). This "scale-disentanglement" allows for precise, unsupervised control over image synthesis, enabling tasks like swapping facial expressions while maintaining identity by simply mixing and matching latent sub-vectors.

What Problem Does This Paper Address?

Unconditional Generative Adversarial Networks (GANs) are excellent at creating realistic images from random noise, but they are notoriously difficult to control. Because the relationship between the input noise vector (the latent space) and the final image is highly complex and entangled, we don't intuitively know which "knob" to turn to change a specific feature, like a person's expression or the color of a car.

While previous work like Progressive Growing of GANs (PGGAN) improved image quality by adding network layers over time, the features themselves remained entangled across the latent space. The authors of BSD-GAN sought to create a "steerable" GAN where different parts of the input noise vector are explicitly responsible for different scales of visual information—coarse structures (like head pose), mid-level features (like face shape), and fine details (like skin pores or hair strands).

Methods

The core innovation is a training pipeline that grows the network in both depth (adding layers) and width (activating new input sub-vectors).

  1. Sub-vector Branching: The input noise vector $z$ is split into several sub-vectors ($z^0, z^1, \dots, z^n$).
  2. Progressive De-freezing: Training starts at a low resolution with only the first sub-vector ($z^0$) active. As the resolution of training images increases, the authors add new layers to the generator and discriminator and "de-freeze" the next sub-vector in the sequence.
  3. Branch Suppression: The authors discovered a key phenomenon: once a branch (like $z^0$) has learned to represent coarse structures, it maintains dominance over them. When a new branch ($z^1$) is introduced, it is "suppressed" from learning those already-encoded coarse features and is instead forced to capture the new, finer-scale details present in the higher-resolution training data.
  4. Smooth Transitions: To prevent "sudden shock" to the network when a new branch is added, they use a two-stage sub-procedure where the influence of the new sub-vector is gradually increased.
Fig. 2. Training pipeline of BSD-GAN. We start with both the generator (G) and discriminator (D) having a low spatial resolution. During the first training period, we feed z0 with random vectors of uniform distribution and zt ( t > 0) with zero vectors 0, and make those linear layers corresponding to zt ( t > 0) untrainable. As the training advances, we incrementally add layers to G and D, thus increasing the spatial resolution of the generated images. Meanwhile, we “de-freeze” more z vectors for training by feeding them with non-zero uniform-random vectors. This process is repeated until the target resolution is reached. During training, “branch suppression” (see Section 3 for details) happens as z0 has well encoded large-scale structures and will maintain its dominance in coarse-level encoding. When z1 is de-frozen, it is suppressed in terms of coarse-level encoding but has the chance to encode finer-scale features.

Datasets

The researchers evaluated BSD-GAN on three high-quality datasets:

  • CelebA_HQ: A standard benchmark for high-resolution face generation.
  • LSUN church_outdoor: Used to test the model's ability to handle complex architectural scenes.
  • Car Dataset: A new dataset introduced by the authors featuring high-resolution (800x600) images of cars, providing a different set of structural challenges compared to faces.

Key Insights and Ideas

  • Exploiting "Branch Suppression": Rather than fighting the tendency of networks to prioritize earlier-learned features, BSD-GAN uses this as a mechanism for disentanglement. It’s a "first-come, first-served" approach to feature learning.
  • Simplicity is Strength: Unlike concurrent work like StyleGAN, which uses a complex style-based architecture, BSD-GAN achieves impressive disentanglement primarily through its training schedule and a simple branching of the input vector.
  • Direct Latent Surgery: Because the sub-vectors are explicitly tied to scales, users can perform "cross-scale image fusion" without any optimization. You can take the "coarse" sub-vector from Person A and the "fine" sub-vectors from Person B to perform a face swap or expression transfer instantly.
Examples of expression transfer and face swap achieved by swapping sub-vectors.

Key Results

  • Quantitative Disentanglement: The authors introduced a metric called Variance By Scale (VBS), which uses Fourier transforms to measure how much a sub-vector affects different frequency ranges. Their results showed clear peaks: $z^0$ dominated low frequencies (coarse shapes), while subsequent sub-vectors peaked at progressively higher frequencies (fine details).
  • Superiority over StyleGAN: In comparative tests, BSD-GAN showed better scale-disentanglement than StyleGAN, where many noise chunks were found to only affect the very finest details or have redundant effects.
  • Enhanced Interactive Editing: When integrated into the iGAN framework (an interactive image editing tool), BSD-GAN allowed for much more accurate fitting to user sketches and color strokes, achieving lower optimization losses than DCGAN or PGGAN.

Limitations and Future Work

  • Scale vs. Semantics: While the model disentangles by scale, it is not yet feature-aware. It cannot, for example, isolate just the "eyes" if other facial features exist at the same scale.
  • Perspective Challenges: The model struggles with features that appear at different scales due to perspective (e.g., a row of fence posts receding into the distance).
  • Future Directions: The authors suggest extending this progressive training paradigm to be "semantic-aware," where sub-vectors could be designated for more meaningful, visually apparent image features rather than just raw scale.

Summary updated

Abstract

We introduce BSD-GAN, a novel multi-branch and scale-disentangled training method which enables unconditional Generative Adversarial Networks (GANs) to learn image representations at multiple scales, benefiting a wide range of generation and editing tasks. The key feature of BSD-GAN is that it is trained in multiple branches, progressively covering both the breadth and depth of the network, as resolutions of the training images increase to reveal finer-scale features. Specifically, each noise vector, as input to the generator network of BSD-GAN, is deliberately split into several sub-vectors, each corresponding to, and is trained to learn, image representations at a particular scale. During training, we progressively "de-freeze" the sub-vectors, one at a time, as a new set of higher-resolution images is employed for training and more network layers are added. A consequence of such an explicit sub-vector designation is that we can directly manipulate and even combine latent (sub-vector) codes which model different feature scales.Extensive experiments demonstrate the effectiveness of our training method in scale-disentangled learning of image representations and synthesis of novel image contents, without any extra labels and without compromising quality of the synthesized high-resolution images. We further demonstrate several image generation and manipulation applications enabled or improved by BSD-GAN. Source codes are available at https://github.com/duxingren14/BSD-GAN.
BibTeX
@misc{yi2018bsd,
  author = {Zili Yi and Zhiqin Chen and Hao Cai and Wendong Mao and Minglun Gong and Hao Zhang},
  title = {{BSD-GAN: Branched Generative Adversarial Network for Scale-Disentangled Representation Learning and Image Synthesis}},
  year = {2018},
  eprint = {1803.08467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.08467},
}

Loading related content...

Figure