Low Distortion Block-Resampling with Spatially Stochastic Networks

Sarah Jane Hong, Martín Arjovsky, Darryl Barnhart, Ian ThompsonNeurIPS 2020

Generative Adversarial Networks (GANs) are unparalleled at creating realistic, high-resolution images, but they typically suffer from an "all-or-nothing" control problem. If a model generates a highly realistic face but the user dislikes the hairstyle, the standard options are either to keep the image as-is or generate a completely new one from scratch.

Traditional inpainting attempts to solve this by rigidly fixing the surrounding pixels and regenerating only a masked area. However, for structured data like human faces or architecture, this strict boundary constraint is often too restrictive and harms realism—a phenomenon the authors term the Inpainting Paradox. For instance, if you change a subject's hairstyle, the forehead or ears might need to shift slightly to maintain a natural, globally consistent look.

To address this, the authors propose Low Distortion Block-Resampling (LDBR). LDBR is a framework where a specific "block" of an image can be resampled freely, while the rest of the image is allowed to change just enough to remain globally consistent, staying as close to the original as possible without rigid pixel-locking.

Spatially Stochastic Networks (SSNs)

To implement LDBR, the authors introduce Spatially Stochastic Networks (SSNs), which rely on three core methodological innovations:

1. Spatial Latent Decomposition

Instead of using a single, flat latent vector $z$ to generate an entire image, the latent code is treated as a 3D spatial tensor. Each "block" of this tensor corresponds to a specific spatial area of the output image.

Spatial Latent Decomposition

By isolating specific blocks (e.g., the blocks corresponding to the towers of a church), the network can "reroll" those specific features while leaving the latent codes for the rest of the image untouched.

2. Spatially Modulated Convolution

The authors observed that existing spatially-conditioned models (like SPADE) often suffer from "droplet" artifacts. Recognizing that these artifacts were identical to those found in the original StyleGAN, they adapted the weight demodulation technique from StyleGAN2 to a spatial context. This new Spatially Modulated Convolution layer provides high-quality, spatially-aware generation while eliminating the droplet artifacts.

3. Distortion Regularization

Simply decomposing the latent space isn't enough; changing one latent block could theoretically cause the entire image to fluctuate wildly. To prevent this, the authors add an explicit penalty term to the loss function. This regularization encourages the generator to minimize changes in image blocks $y_b$ when a different, non-corresponding latent block $z_a$ (where $a \neq b$) is resampled.

Training Efficiency and Datasets

The model was evaluated on two standard high-resolution ($256 \times 256$) benchmarks: FFHQ (human faces) and LSUN Churches (outdoor architecture).

Notably, the team utilized transfer learning from pretrained StyleGAN2 models rather than training from scratch. They justified this with a striking environmental and financial argument: training from scratch would have cost approximately $400,000 and emitted 20,000 kg of $CO_2$. In contrast, their transfer learning protocol cost only $2,000 and emitted a mere 0.36 kg of $CO_2$.

Key Results and Insights

  • Quality vs. Control: Adding spatial control did not degrade image quality. In fact, SSNs achieved a Frechet Inception Distance (FID) of 12.24 on FFHQ, significantly outperforming the StyleGAN2 baseline (19.76) used in their tests.
  • Distortion Reduction: By tuning the regularization parameter $\lambda_D$, the authors successfully reduced image distortion by an entire order of magnitude (from 0.028 to 0.0043) with only a minor impact on overall image quality.
  • Smoother Latent Space: Surprisingly, Perceptual Path Length (PPL)—a metric measuring the smoothness of the latent space—actually improved as distortion regularization increased. Forcing the network to spatially disentangle features seemingly makes the latent space more organized.
Qualitative Results on FFHQ

Qualitatively, the network excels at localized edits. Users can change a person's hair color, alter a smile, or swap out architectural features of a church while the rest of the scene remains remarkably stable. The difference maps clearly show that modifications are localized to the targeted semantic regions while preserving global identity.

Limitations and Future Work

While SSNs offer a massive leap in spatial control, the authors note a few limitations:

  1. Distortion Metric: The current distortion penalty relies on Mean Squared Error (MSE) in pixel space, which the authors acknowledge is a poor proxy for human perception. Future iterations might use Earth Mover’s Distance or feature-space distances.
  2. Fixed Grids: The resampling blocks are currently pre-specified rectangular grids. Moving toward semantic blocks—allowing users to select "the eyes" or "the trees" via a segmentation map—would provide much more intuitive control.
  3. Single Resolution: The spatial latent code operates at a single resolution. Developing multi-scale spatial latents could allow for fine-grained control over both coarse structural elements and fine textures simultaneously.

Summary updated

BibTeX
@inproceedings{hong2020low,
  author = {Sarah Jane Hong and Martín Arjovsky and Darryl Barnhart and Ian Thompson},
  title = {{Low Distortion Block-Resampling with Spatially Stochastic Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure