Elevating Perceptual Sample Quality in PCs through Differentiable Sampling

Steven Lang, Martin Mundt, Fabrizio Ventola, Robert Peharz, Kristian KerstingPre-Registration Workshop @ NeurIPS 2021 · OpenReview

Overview

This paper introduces a novel differentiable sampling procedure for Probabilistic Circuits (PCs), allowing them to be trained with "perceptual" losses typically reserved for deep generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). While the authors found that matching the sample quality of neural networks is harder than initially hypothesized, their work successfully bridges the gap between tractable probabilistic modeling and modern deep learning optimization, enabling PCs to generate samples with significantly richer detail and contrast.

The Problem: Tractability vs. Sample Quality

In the world of generative modeling, there is a well-known trade-off between tractability and sample quality.

Probabilistic Circuits (PCs), such as Sum-Product Networks, excel at tractability. They allow for exact and efficient answers to complex probabilistic queries (like marginalization or conditioning) that are computationally intractable for standard neural networks. However, PCs are traditionally trained using Maximum Likelihood Estimation (MLE). While MLE is mathematically sound, it often results in "blurry" or "average-looking" samples because it does not account for human perceptual judgment.

On the other hand, Deep Generative Models (GANs, VAEs) produce stunning samples by using alternative loss functions that compare generated samples directly to real data. The challenge is that these models require differentiable sampling—the ability to pass a gradient from the output sample back through the sampling process to update the model parameters. Because sampling in a PC involves discrete choices (e.g., selecting which branch of a sum node to follow), it is inherently non-differentiable, historically leaving PCs stuck in the MLE era.

Methods: Making Sampling Differentiable

The authors propose a method to make PC sampling "smooth" and differentiable using key reparameterization techniques.

Comparison of standard non-differentiable sampling (a) and the proposed differentiable sampling using the Gumbel-Softmax trick (b).
  1. The Gumbel-Softmax Trick: To handle the discrete choice at "sum nodes" (which act like mixture models), they replace the categorical draw with a continuous approximation. This allows a "reparameterization" where the randomness is moved to an external variable, making the path from parameters to samples differentiable.
  2. Smooth Child Interpolation: In standard PC sampling, a single "child" node is selected to follow. To maintain differentiability, the authors instead sample from all children and combine their outputs using a weighted average (a dot product) based on the Gumbel-Softmax weights.
  3. Leaf Node Reparameterization: They also address the "leaf nodes" (the base distributions). For example, they approximate Binomial distributions with Gaussians to enable the use of the standard reparameterization trick.

By combining these techniques, PCs can be trained using three modern objectives:

  • Adversarial Training (ADV): Using a discriminator to distinguish "real" from "fake" samples, similar to a GAN.
  • Maximum Mean Discrepancy (MMD): Minimizing the distance between the distributions of real and generated samples in a kernel space.
  • Probabilistic Auto-Encoding (PAE): Training the PC as an encoder-decoder structure, similar to a VAE.

Datasets and Evaluation

The researchers evaluated their approach on four standard image benchmarks: MNIST (handwritten digits), SVHN (Street View House Numbers), CIFAR-10 (general objects), and CelebA (celebrity faces). They also utilized six synthetic 2D datasets (such as 2-moons and 9-clusters) to visualize how different loss functions alter the way a PC shapes a probability distribution.

Key Insights

  • The "Early GAN" Parallel: The authors discovered that training PCs with adversarial losses is just as finicky as training early GANs. They encountered familiar issues like mode collapse (where the model generates the same image repeatedly) and training instability.
  • Contrast over Likelihood: While MLE-trained PCs often achieve better "likelihood" scores, the perceptually-trained PCs produced images with much higher contrast and sharper details. This confirms that mathematically "good" probability doesn't always translate to visually "good" samples.
  • Leaf Distributions Matter: A surprising insight was that the choice of leaf nodes (the very bottom of the circuit) significantly impacts the success of differentiable sampling. Moving from simple Gaussians to a Gaussian-approximated Binomial leaf vastly improved results on image data.
  • The Hypothesis Check: The authors' initial hypothesis—that perceptual losses would immediately put PCs on par with GANs—was rejected. PCs are currently at a stage similar to where deep neural networks were a decade ago; they require more specialized architectures and engineering tricks to fully close the gap.

Key Results

  • Qualitative Improvement: Visual inspections demonstrate that ADV and MMD objectives produce samples with richer colors and more distinct features compared to the "washed-out" samples of standard MLE training.
  • Quantitative Gap: In terms of FID (Fréchet Inception Distance)—a standard measure of image quality—the PCs still lag behind deep neural networks. For CelebA, the PC achieved an FID of ~354-420, while a comparable VAE/GAN scored ~316-358 (lower is better).
  • Synthetic Success: On 2D data, the ADV objective effectively captured the "modes" (clusters) of data, though it sometimes struggled with complex non-linear shapes compared to standard MLE.

Limitations and Future Work

The authors are transparent about the ongoing challenges. Differentiable sampling in PCs is difficult to optimize, and the models are prone to the same failure modes as vanilla GANs and VAEs. Furthermore, they note that standard metrics like FID might not perfectly capture the subjective quality of PC samples at this stage.

For future directions, they suggest hybrid training: pre-training a PC with MLE to establish a stable foundation, then fine-tuning with perceptual losses to "sharpen" the samples. They also call for more research into optimizing "leaf" distributions specifically suited for this new differentiable regime.

Summary updated

Abstract

Deep generative models have seen a dramatic improvement in recent years, due to the use of alternative losses based on perceptual assessment of generated samples. This improvement has not yet been applied to the model class of probabilistic circuits (PCs), presumably due to significant technical challenges concerning differentiable sampling, which is a key requirement for optimizing perceptual losses. This is unfortunate, since PCs allow a much wider range of probabilistic inference routines than main-stream generative models, such as exact and efficient marginalization and conditioning. Motivated by the success of loss reframing in deep generative models, we incorporate perceptual metrics into the PC learning objective. To this aim, we introduce a differentiable sampling procedure for PCs, where the central challenge is the non-differentiability of sampling from the categorical distribution over latent PC variables. We take advantage of the Gumbel-Softmax trick and develop a novel inference pass to smoothly interpolate child samples as a strategy to circumvent non-differentiability of sum node sampling. We initially hypothesized, that perceptual losses, unlocked by our novel differentiable sampling procedure, will elevate the generative power of PCs and improve their sample quality to be on par with neural counterparts like probabilistic auto-encoders and generative adversarial networks. Although our experimental findings empirically reject this hypothesis for now, the results demonstrate that samples drawn from PCs optimized with perceptual losses can have similar sample quality compared to likelihood-based optimized PCs and, at the same time, can express richer contrast, colors, and details. Whereas before, PCs were restricted to likelihood-based optimization, this work has paved the way to advance PCs with loss formulations that have been built around deep neural networks in recent years.
BibTeX
@inproceedings{lang2021elevating,
  author = {Steven Lang and Martin Mundt and Fabrizio Ventola and Robert Peharz and Kristian Kersting},
  title = {{Elevating Perceptual Sample Quality in PCs through Differentiable Sampling}},
  booktitle = {Pre-Registration Workshop @ NeurIPS},
  year = {2021},
  url = {https://openreview.net/forum?id=VxTV5nWG96},
}
Figure