Discrete Distribution Networks

Lei Yang

Introduction

Modeling complex, high-dimensional data distributions remains a central challenge in generative AI. While Diffusion models and Generative Adversarial Networks (GANs) have achieved significant success, they often require expensive iterative sampling or face training instabilities. Furthermore, achieving Zero-Shot Conditional Generation (ZSCG)—steering a model toward a new condition (like a text prompt or sketch) not seen during training—typically necessitates retraining or complex gradient-based optimization.

Discrete Distribution Networks (DDN) propose a shift in this paradigm by treating generation as a sequence of discrete selections from an exponentially expanding search space, enabling gradient-free steering and efficient hierarchical modeling.

Methods: The Discrete Distribution Layer

The fundamental building block of DDN is the Discrete Distribution Layer (DDL). Unlike standard layers that produce a single continuous output, each DDL outputs $K$ distinct candidate images.

Overview of the DDN architecture and the Discrete Distribution Layer (DDL).

Hierarchical Generation and Guided Sampling

Generation proceeds through $L$ layers. During training, a "Guided Sampler" identifies the candidate closest to the ground truth (using L2 distance). This selection is concatenated with the input feature and passed to the next layer to produce $K$ more refined candidates. This process effectively builds a path through a tree of possibilities.

The "Split-and-Prune" Algorithm

To ensure the discrete candidates accurately represent the data distribution, DDN employs an evolutionary optimization strategy. If an output node is frequently selected, it "splits" (clones itself) to explore finer details. Conversely, rarely chosen "dead nodes" are "pruned." This mechanism keeps the discrete samples aligned with the actual density of the training data.

Key Insights

Gradient-Free Steering

Because DDN provides $K$ explicit choices at every step, it can be guided by "black-box" models where gradients are unavailable. For conditional generation, the training-time Guided Sampler is replaced by a discriminative model (e.g., CLIP or a classifier). The external model simply selects the candidate that best satisfies the condition at each layer, allowing for zero-shot synthesis without backpropagation through the guiding model.

Tree-Structured Latent Space

DDN represents an image as a sequence of integers—the indices of the chosen candidates at each layer. This creates a 1D latent representation corresponding to a path down a massive tree. With $L$ layers and $K$ choices, the model can represent $K^L$ possible samples, providing an enormous search space for diversity.

Experimental Results

DDN was evaluated on standard benchmarks including MNIST, CIFAR-10, FFHQ, and CelebA-HQ.

Zero-shot conditional generation results across various tasks including super-resolution, inpainting, and CLIP-guided synthesis.

Zero-Shot Versatility

The model demonstrates remarkable flexibility, performing tasks such as super-resolution, colorization, inpainting, and text-to-image synthesis without task-specific training. As shown in the qualitative results, DDN can adapt to various degradations and semantic constraints (like adding "sunglasses" via CLIP) purely through guided selection at inference time.

Quantitative Performance

On CIFAR-10, DDN achieved an FID of 52.0, outperforming VAEs (106.7) and Gated PixelCNN (65.9). While it currently trails state-of-the-art GANs and Diffusion models in raw FID, its unique ability to perform gradient-free conditional generation offers a distinct advantage in flexibility.

Limitations and Future Work

While promising, DDN faces some challenges. Performing ZSCG requires $L \times K$ forward passes through the guiding model, which can be computationally intensive. Additionally, the reliance on L2 loss during training can lead to slightly blurred textures in high-resolution outputs. Future work may involve incorporating adversarial losses to improve fine-grained detail and exploring "Loss-Free Balancing" to scale the model more efficiently without discarding parameters during pruning.

Summary updated

BibTeX
@inproceedings{yang2025discrete,
  author = {Lei Yang},
  title = {{Discrete Distribution Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=xNsIfzlefG},
}
Figure