Glow: Generative Flow with Invertible 1x1 Convolutions

Diederik P. Kingma, Prafulla Dhariwal

Generative modeling—the art of teaching a machine to create new data that mimics its training set—has traditionally been dominated by three approaches, each with significant trade-offs. Autoregressive models (like PixelCNN) produce high-quality images but are painfully slow due to pixel-by-pixel generation. Variational Autoencoders (VAEs) are fast but optimize a lower bound of the data's likelihood, often yielding blurry results. Generative Adversarial Networks (GANs) produce sharp images but are notoriously difficult to train, lack an encoder to map images back to latent space, and suffer from mode collapse.

Flow-based models offer an elegant alternative. By using a series of invertible transformations, they map complex data distributions (like human faces) to simple ones (like a Gaussian). While theoretically appealing due to exact latent-variable inference and exact log-likelihood evaluation, earlier flow models like RealNVP struggled to produce high-resolution images.

Enter Glow, a flow-based generative model that introduces invertible $1 \times 1$ convolutions to simplify and dramatically improve the performance of normalizing flows, enabling the synthesis of remarkably realistic, high-resolution images.

The Architecture of Glow

Glow builds upon the normalizing flow framework, where an image is transformed into a latent code through a sequence of invertible steps. The paper introduces a novel "step of flow" consisting of three core components:

  1. Actnorm (Activation Normalization): A specialized layer that performs an affine transformation (scale and bias) per channel. Using data-dependent initialization, it ensures the model starts with zero mean and unit variance. This solves training stability issues typically addressed by Batch Normalization, which performs poorly with the small batch sizes required for high-resolution images.
  2. Invertible $1 \times 1$ Convolution: This is Glow's primary innovation. Previous models used fixed, hard-coded permutations to mix information across channels. Glow replaces these with a learned $1 \times 1$ convolution. To maintain computational efficiency, the authors parameterize the weight matrix using LU decomposition, reducing the cost of calculating the determinant from $O(c^3)$ to $O(c)$.
  3. Affine Coupling Layers: A standard technique inherited from RealNVP, these layers allow for powerful non-linear transformations while remaining easy to invert and possessing a simple Jacobian determinant.

These steps are stacked in a multi-scale architecture, allowing the model to learn features at various resolutions.

Key Capabilities and Insights

Glow's architecture unlocks several powerful capabilities for generative modeling:

  • Learned Permutations Matter: Replacing fixed channel shuffling with learned $1 \times 1$ convolutions significantly speeds up convergence and results in a better (lower) negative log-likelihood.
  • Exactness is a Superpower: Because the model is fully invertible, users can take a real image, find its exact latent representation, modify it, and map it back to the pixel space. There is no approximation error, unlike in VAEs.
  • Semantic Latent Space: The latent space in Glow is highly structured. By calculating the average latent vector for specific attributes (e.g., "smiling" vs. "not smiling"), one can perform "vector arithmetic" on faces. Adding a "smile vector" to a neutral face's latent code reliably produces a smiling version of that exact person.
  • Scaling to High-Resolution: Glow was the first flow-based model to demonstrate that optimizing for pure log-likelihood—without the adversarial loss of a GAN—could produce crisp, realistic $256 \times 256$ images.

Performance and Results

The authors evaluated Glow on standard benchmarks (CIFAR-10, ImageNet, LSUN) and the high-resolution CelebA-HQ dataset ($256 \times 256$ celebrity faces).

Quantitative Improvements: Glow consistently outperformed its predecessor, RealNVP, across all datasets in terms of bits per dimension (BPD). On CIFAR-10, Glow achieved a highly competitive 3.35 BPD compared to RealNVP's 3.49.

Qualitative Fidelity: Perhaps most strikingly, the model produced incredibly high-fidelity synthetic faces, proving that flow-based models can compete with GANs in visual quality.

High-fidelity synthetic faces generated by Glow

Computational Efficiency: Despite the architectural complexity, synthesis is highly parallelizable. Generating a $256 \times 256$ image takes less than 130ms on a single consumer-grade GPU (GTX 1080 Ti).

Limitations and Future Directions

While Glow represents a massive leap for flow-based models, it is not without limitations. Flow models are inherently memory-intensive because they require storing many intermediate activations to compute gradients. Although the authors utilized gradient checkpointing, the memory footprint remains a hurdle for extremely deep models.

Additionally, to train on $256 \times 256$ images within memory constraints, the color depth was reduced to 5-bit images, slightly impacting color nuance. Finally, training these models to convergence on high-resolution data requires significant compute resources compared to smaller GANs.

Future work may leverage the model's reversibility to achieve "constant memory" training—where memory usage does not scale linearly with model depth—further unlocking the potential of flow-based generative models.

Summary updated

Abstract

Flow-based generative models are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using invertible 1x1 convolution. Using our method we demonstrate a significant improvement in log-likelihood and qualitative sample quality. Perhaps most strikingly, we demonstrate that a generative model optimized towards the plain log-likelihood objective is capable of efficient synthesis of large and subjectively realistic-looking images.
BibTeX
@inproceedings{kingma2018glow,
  author = {Diederik P. Kingma and Prafulla Dhariwal},
  title = {{Glow: Generative Flow with Invertible 1x1 Convolutions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=ByZWJFWuZr},
}
Figure