Structured Output Learning with Conditional Generative Flows

You Lu, Bert HuangAAAI 2020 · DOI

Introduction

Structured prediction is a fundamental challenge in machine learning, involving the mapping of an input $x$ (such as an image) to a high-dimensional, highly correlated output $y$ (such as a segmentation mask or a denoised image). Capturing the complex dependencies between the dimensions of $y$ typically requires sophisticated modeling techniques.

Traditionally, researchers have turned to Energy-Based Models (EBMs) or Conditional Random Fields (CRFs). While powerful, these models suffer from a significant drawback: the "partition function" problem. Calculating the probability $p(y|x)$ requires computing a normalization constant that involves an intractable integral over all possible outputs in high-dimensional space. This makes training difficult, often necessitating complex approximations, surrogate objectives, or computationally expensive iterative inference.

To address these challenges, this paper introduces Conditional Glow (c-Glow), a conditional version of the Glow generative flow model. By leveraging the invertibility of normalizing flows, c-Glow enables the exact and efficient computation of conditional log-likelihoods, bypassing the intractability of traditional structured prediction models.

Architecture and Methods

The core of c-Glow is the "Glow" architecture, a type of normalizing flow that uses a sequence of invertible transformations to map a complex data distribution to a simple latent space, such as a Gaussian. The key innovation in c-Glow is the systematic incorporation of conditioning information from the input $x$ into the flow's core components.

Comparison of the standard Glow architecture and the proposed conditional c-Glow architecture.

As illustrated in the comparison above, the authors modify three primary layers to create the conditional flow:

  1. Conditional Actnorm: Instead of using fixed parameters, a "Conditioning Network" (CN) dynamically generates scale and bias parameters based on the input $x$.
  2. Conditional 1×1 Convolution: The weight matrix used for channel permutation is generated by a CN from the input $x$, allowing the permutation to be data-dependent.
  3. Conditional Affine Coupling: Features extracted from $x$ are concatenated with the hidden state, determining the scaling and translation of the output in a way that is directly informed by the input context.

Training and Inference

Because c-Glow is a normalizing flow, the change-of-variables formula allows for the calculation of the exact log-likelihood of the data. Training is straightforward: the model parameters are optimized to maximize this likelihood using standard gradient descent.

For inference—finding the optimal output $y^*$—the authors discovered that standard gradient-based optimization was often slow and susceptible to local optima. Instead, they proposed a sample-based approach. By drawing multiple samples from the latent space and passing them through the inverse flow, they generate a set of candidate outputs. Averaging these candidates provides a "smoothing" effect that leads to more robust and accurate predictions than single-sample or gradient-based methods.

Experimental Results

The authors evaluated c-Glow across five diverse structured prediction tasks, demonstrating its versatility and performance.

  • Binary Segmentation: On the Weizmann Horse database ($128 \times 128$), c-Glow achieved an Intersection over Union (IOU) of 0.858, significantly outperforming a standard Feed-forward Convolutional Network (FCN) which scored 0.795.
Example of a binary segmentation mask produced for the Weizmann Horse dataset.
  • Image Inpainting: Using the CelebA dataset, c-Glow achieved a PSNR of 24.88, surpassing the state-of-the-art DCGAN inpainting method (22.73) and its Poisson-blended variant (23.65).
  • Depth Refinement: On the Seven Scenes dataset, c-Glow (36.53 PSNR) slightly outperformed ProximalNet (36.31), a specialized deep structured model.
  • Multi-class Segmentation and Denoising: The model also showed competitive results on LFW face segmentation and BSDS500 color image denoising.

Key Insights and Discussion

A major takeaway from this work is that exact likelihood optimization provides a more stable and effective training objective than the approximations required by EBMs or SPENs. Furthermore, the model's versatility is notable; the same basic architecture handles both discrete tasks (via uniform noise dequantization) and continuous tasks with minimal adjustments.

The "Power of Averaging" was another critical finding. The sample-based inference method helped the model avoid "hallucinating" incorrect features. For instance, in segmentation tasks, individual samples might occasionally produce anomalous features (like a "horse horn" effect), but averaging 10 samples effectively smooths out these errors.

However, some limitations remain. In denoising tasks, c-Glow occasionally produced checkerboard artifacts, likely due to the "squeeze" layers in the flow architecture. Additionally, while sampling is faster than iterative optimization, generating and averaging multiple samples still incurs a higher computational cost than a simple feed-forward pass in non-structured models. Future work may involve exploring more sophisticated conditioning networks, such as Transformers, to better capture global context.

Summary updated

BibTeX
@inproceedings{lu2020structured,
  author = {You Lu and Bert Huang},
  title = {{Structured Output Learning with Conditional Generative Flows}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5940},
}
Figure