Probabilistic Neural Programmed Networks for Scene Generation

Zhiwei Deng, Jiacheng Chen, Yifang Fu, Greg MoriNeurIPS 2018 · OpenReview

TL;DR

PNP-Net is a modular generative framework that uses "neural programs" to compose complex images from text descriptions. By treating visual concepts as latent distributions and combining them through specialized probabilistic operators, the model achieves superior compositionality and zero-shot generalization compared to standard end-to-end generative models.

High-level overview of the PNP-Net framework.

What Problem Does This Paper Address?

Generating a single object (like a face or a handwritten digit) is a well-studied problem, but generating complex scenes—composed of multiple objects with specific attributes and spatial relationships—remains a major challenge.

Standard generative models like Variational Auto-Encoders (VAEs) and Generative Adversarial Networks (GANs) often struggle with "compositionality." For example, if a model learns "red cube" and "blue sphere," it might fail to generate a "blue cube" if that specific combination wasn't in the training set. Furthermore, these models often lack a structured way to handle spatial relations (e.g., "to the left of") or varying numbers of objects. The authors aim to bridge the gap between powerful pixel-level decoders and structured, programmatic latent representations.

Methods

The authors propose PNP-Net, which integrates modular programmable networks into a VAE framework. Instead of a single "black box" encoder/decoder, the generation process follows a tree structure derived from a text-based program.

Key Components

The model represents concepts (like "shiny" or "cylinder") as Gaussian distributions in a latent space. To manipulate these distributions, PNP-Net utilizes four core "operators":

The four core latent distribution operators used in PNP-Net.
  • Combine: Merges attributes (e.g., "shiny" + "red") using a Gated Product of Experts (g-PoE). This allows the model to selectively combine information from different semantic experts while avoiding artifacts from overlapping distributions.
  • Describe: Models the interaction between an attribute and an object (e.g., "red" + "sphere") in a content-aware manner.
  • Transform: Handles scale invariance by sampling bounding box sizes and performing bilinear interpolation directly on the latent distributions.
  • Layout: Arranges objects on a canvas based on spatial relations (e.g., "left-front"), refining the final latent map through a convolutional neural network (CNN).

Additionally, the model utilizes a Dual Latent Space. The latent representation $z$ is split into $z_a$ (appearance) and $z_s$ (location/scale), allowing the model to disentangle what an object looks like from where it is. The entire framework is trained by maximizing the Evidence Lower Bound (ELBO), with auxiliary losses for bounding box and offset samplers to improve spatial accuracy.

Datasets

The researchers evaluated PNP-Net on three primary benchmarks:

  1. ColorMNIST: A custom dataset featuring images with up to two MNIST digits, varying in size (small/large), color (6 options), and spatial relationship (top, bottom, left, right).
  2. CLEVR-G: A generative version of the CLEVR dataset containing 10,000 images (64x64) with objects of different sizes, materials, colors, and six spatial relations.
  3. CLEVR-G-ZS (Zero-Shot): A challenging split where certain combinations of colors, materials, and objects are withheld during training to test the model's ability to generalize to novel scenes.

Key Insights and Ideas

  • Programs over Pixels: By moving the "logic" of the scene (composition and layout) into the latent space via operators, the decoder only needs to focus on rendering those latent codes into realistic pixels.
  • The Power of g-PoE: Standard Product of Experts can suffer from "ghost information" (artifacts from overlapping distributions). The gated version (g-PoE) introduced here allows the model to "shut off" irrelevant information during the combination of attributes.
  • Zero-Shot Generalization: Because the operators (like "red" or "left-of") are learned as modular functions, the model can successfully generate a "red shiny cylinder" even if it has never seen that specific combination, provided it has seen those individual components elsewhere.

Key Results

The authors used mean Average Precision (mAP) from a pre-trained detector to measure "semantic correctness"—essentially asking a separate AI if the generated image actually contains what it was supposed to.

  • Semantic Accuracy: On the CLEVR-G dataset (4-object scenes), PNP-Net significantly outperformed baselines. For "Object-Attribute" (OBJ-A) accuracy, PNP-Net achieved 0.737, while DCGAN and PixelCNN trailed at 0.176 and 0.074, respectively.
  • Zero-Shot Performance: In the zero-shot setting, PNP-Net maintained a high OBJ-A score of 0.734, demonstrating that its modular nature allows it to "understand" concepts independently of their training-time contexts.
  • Complexity Scaling: The model proved robust when scaling from 4-object training scenes to 8-object test scenes, outperforming baselines that struggled with the increased density and occlusion.

Limitations and Future Work

  • Program Dependency: The model currently relies on a pre-defined tree structure (program) derived from text. While the authors suggest this can be automated, the current results assume the program is given.
  • Resolution: The experiments were conducted at a relatively low resolution (64x64). Scaling this modular approach to high-resolution, photorealistic scenes (like COCO) would likely require more complex decoders and more sophisticated layout operators.
  • Supervision: The training process uses ground-truth bounding boxes as an auxiliary loss, which is a form of strong supervision that might not be available for all datasets. Future work could explore learning these spatial operators in a more unsupervised or weakly-supervised manner.

Summary updated

Abstract

In this paper we address the text to scene image generation problem. Generative models that capture the variability in complicated scenes containing rich semantics is a grand goal of image generation. Complicated scene images contain rich visual elements, compositional visual concepts, and complicated relations between objects. Generative models, as an analysis-by-synthesis process, should encompass the following three core components: 1) the generation process that composes the scene; 2) what are the primitive visual elements and how are they composed; 3) the rendering of abstract concepts into their pixel-level realizations. We propose PNP-Net, a variational auto-encoder framework that addresses these three challenges: it flexibly composes images with a dynamic network structure, learns a set of distribution transformers that can compose distributions based on semantics, and decodes samples from these distributions into realistic images.
BibTeX
@inproceedings{deng2018probabilistic,
  author = {Zhiwei Deng and Jiacheng Chen and Yifang Fu and Greg Mori},
  title = {{Probabilistic Neural Programmed Networks for Scene Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=H1-MWtbdbr},
}
Figure