Discrete Distribution Networks

Lei Yang

Modern generative models, such as diffusion models and GANs, are incredibly powerful but often require training separate models or parameters for specific conditional generation tasks (e.g., text-to-image, inpainting). While Zero-Shot Conditional Generation (ZSCG) exists, current methods usually require conditions to be in the same pixel domain as the training data, or they rely heavily on calculating gradients from discriminative models during the generation process.

To address this, researchers have introduced Discrete Distribution Networks (DDN), a novel generative model that approximates complex data distributions through hierarchical discrete sampling. By generating multiple samples per layer and iteratively refining the best match, DDN creates an exponentially large representation space. This unique architecture enables powerful, gradient-free zero-shot conditional generation and highly compressed 1D latent representations.

How DDN Works

The DDN architecture flips the standard neural network paradigm. Instead of producing a single output, each layer in a DDN generates $K$ distinct samples.

During training, the model operates in a coarse-to-fine manner. It selects the sample closest (via L2 distance) to the ground truth image. This selected sample is then fed as a condition into the next layer. Over $L$ layers, the representational space grows exponentially to $K^L$.

Overview of the DDN image reconstruction process and its corresponding tree-structured latent space.

To optimize this unique structure, the authors employ a few clever techniques:

  • Split-and-Prune Optimization: To prevent "dead nodes" (outputs that are never selected) and handle probability density shifts, the authors use an evolutionary-inspired algorithm. Frequently used nodes are "split" (cloned and slightly perturbed), while unused nodes are "pruned" (deleted).
  • Chain Dropout: To prevent the model from overfitting on specific hierarchical paths, the model occasionally replaces the "closest match" selection with a random choice during training.
  • Inference & ZSCG: For random generation, the model simply picks randomly at each layer. For Zero-Shot Conditional Generation, the L2 distance metric is swapped out for a black-box evaluator (like a CLIP model or a classifier). The model simply picks the output that scores highest according to the evaluator, allowing the model to be steered without backpropagation.

Key Insights

  • Gradient-Free Steering: Because DDN generates a discrete set of options at each layer, you can steer generation using any scoring function (like a pre-trained ResNet or CLIP) just by picking the highest-scoring option. No gradients are required, making it highly compatible with black-box reward models.
  • Tree-Structured Latent Space: The latent representation is literally the sequence of indices chosen at each layer (a path down a tree, such as the "3-1-2" path shown in the figure above). This makes the latent space highly interpretable and compressible into a 1D sequence of integers.
  • The "Taiji" Connection: If $K=2$, the model makes a binary choice at each layer, effectively compressing an image into a semantically meaningful binary string. The authors playfully call this "Taiji-DDN" after the ancient Chinese philosophy of binary generation.
  • Evolutionary Optimization Works for Neural Nets: The Split-and-Prune method elegantly solves the "dead codebook" problem (commonly found in VQ-VAEs) by treating output nodes like species undergoing natural selection.

Results and Capabilities

DDN was evaluated across several datasets, including CIFAR-10, FFHQ, CelebA-HQ, and MNIST.

In terms of image quality, DDN achieved an FID score of 35.4 on CelebA-HQ (64x64) and 43.1 on FFHQ (64x64). On CIFAR-10, it achieved an FID of 52.0, outperforming older baselines like GLOW and VQ-VAE, though it does not yet beat modern state-of-the-art diffusion models.

Zero-Shot Conditional Generation results, including super-resolution, inpainting, and CLIP-guided editing.

Where DDN truly shines is in its Zero-Shot Generation. The model successfully demonstrated ZSCG using a black-box CLIP model for text-to-image steering on a model trained only unconditionally on FFHQ. For example, it can successfully apply a "wearing sunglasses" prompt to an image without any task-specific fine-tuning.

Furthermore, DDN achieves extreme data compression. A default DDN ($K=512, L=128$) compresses an image into just 1,152 bits while retaining the core semantic structure of the image.

Limitations and Future Directions

While promising, DDN has several limitations that pave the way for future research:

  • Capacity Limits: The $K^L$ output space, while large, may still struggle to represent highly complex, high-resolution distributions. Moving to a patch-based approach could exponentially increase this space.
  • Parameter Waste: The Split-and-Prune algorithm discards trained parameters, which is inefficient for scaling. Future work will explore "Loss-Free Balancing" to maintain node usage without deleting weights.
  • High-Frequency Detail Loss: The heavy compression and reliance on L2 loss cause the model to lose fine, high-frequency details, resulting in slightly blurry images. Incorporating adversarial loss (similar to VQ-GAN) is a proposed fix.
  • Inference Cost for ZSCG: Zero-shot generation requires $L \times K$ forward passes through the guiding model (e.g., CLIP), which creates a significant computational bottleneck.

Overall, Discrete Distribution Networks offer a refreshing, autoregressive-inspired alternative to continuous generative models, opening new doors for highly compressed latent spaces and flexible, gradient-free conditional generation.

Summary updated

Abstract

We introduce a novel generative model, the Discrete Distribution Networks (DDN), that approximates data distribution using hierarchical discrete distributions. We posit that since the features within a network inherently capture distributional information, enabling the network to generate multiple samples simultaneously, rather than a single output, may offer an effective way to represent distributions. Therefore, DDN fits the target distribution, including continuous ones, by generating multiple discrete sample points. To capture finer details of the target data, DDN selects the output that is closest to the Ground Truth (GT) from the coarse results generated in the first layer. This selected output is then fed back into the network as a condition for the second layer, thereby generating new outputs more similar to the GT. As the number of DDN layers increases, the representational space of the outputs expands exponentially, and the generated samples become increasingly similar to the GT. This hierarchical output pattern of discrete distributions endows DDN with unique properties: more general zero-shot conditional generation and 1D latent representation. We demonstrate the efficacy of DDN and its intriguing properties through experiments on CIFAR-10 and FFHQ. The code is available at https://discrete-distribution-networks.github.io/
BibTeX
@inproceedings{yang2025discrete,
  author = {Lei Yang},
  title = {{Discrete Distribution Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=xNsIfzlefG},
}
Figure