Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling

Tianqi Chen, Mingyuan Zhou

Introduction

Denoising Diffusion Probabilistic Models (DDPMs) have established a new paradigm in generative modeling, particularly for continuous data like images. By learning to reverse a Gaussian noise process, these models can synthesize high-fidelity samples. However, the reliance on additive Gaussian noise presents a fundamental mismatch for many real-world data types. Data such as word counts in documents, RNA-sequencing counts, or heavy-tailed physical measurements are non-negative, often discrete, and highly sparse. For these distributions, the continuous, symmetric nature of Gaussian noise fails to respect the "hard floor" of zero or the extreme skewness of the data.

In this paper, we introduce Learning to Jump (JUMP), a generative framework designed to handle the complexities of sparse and heavy-tailed non-negative data. By shifting from a "noise-and-denoise" cycle to a "thinning-and-thickening" process based on Poisson distributions, JUMP provides a mathematically principled approach to generating discrete and non-negative continuous data.

The Limitations of Gaussian Diffusion

Standard diffusion models operate by gradually adding Gaussian noise until the data distribution converges to a standard normal distribution. While effective for images, this approach has three primary drawbacks for count-based or sparse data:

  1. Sparsity Handling: Gaussian models struggle to produce exact zeros. They often generate small positive or negative values (e.g., 0.001), failing to capture the true sparsity patterns inherent in datasets like Bag-of-Words (BOW).
  2. Non-Negativity: Gaussian noise is defined on the entire real line. Forcing it to model strictly non-negative data requires ad-hoc clipping or transformations that can distort the underlying distribution.
  3. Heavy Tails: Many scientific and textual datasets follow power-law or heavy-tailed distributions. Gaussian noise, which decays exponentially, is a poor surrogate for the large-magnitude jumps seen in these data types.

The JUMP Framework

The JUMP recipe replaces Gaussian math with a discrete-aware Poisson logic through a four-step process.

1. Poisson Data Randomization

To bridge the gap between continuous non-negative data and discrete counts, JUMP first converts the input into "latent counts" using a Poisson distribution. A scaling parameter, $\lambda$, serves as a hyperparameter to control the signal-to-noise ratio, allowing the model to handle both discrete integers and continuous values within a unified framework.

2. Forward Process: Binomial Thinning

Instead of adding noise, the forward process involves "thinning" the data. Conceptually, if the data is represented as a collection of discrete units (like pebbles in a bucket), thinning is the process of randomly removing these units until the state reaches zero. Mathematically, this is modeled as Binomial thinning, where zero acts as an absorbing state.

3. Reverse Process: Shifted-Poisson Thickening

The generative (reverse) process starts from the absorbing state of zero and learns to "jump" to higher counts. This "thickening" is achieved through a Shifted-Poisson Jump process. At each discrete time step, a deep neural network is trained to predict the number of units to add back to the current state to eventually reconstruct the original data distribution.

4. Optimization via Bregman Divergence

Standard diffusion models typically minimize a Mean Squared Error ($L_2$ loss). However, $L_2$ loss is optimal for Gaussian distributions. JUMP utilizes a loss function based on Bregman divergence, specifically relative entropy (Kullback-Leibler divergence). This choice is mathematically consistent with the Poisson nature of the thinning/thickening process and is more effective at capturing the nuances of count data.

Experimental Evaluation

The authors evaluated JUMP across synthetic benchmarks, high-dimensional text data, and standard image datasets.

Synthetic Distributions

JUMP was tested on nine univariate distributions, including Poisson and Negative Binomial mixtures, as well as heavy-tailed distributions like the Half-Cauchy. In cases where values can be extremely large (the "long tail"), JUMP accurately captured the distribution, whereas standard DDPMs failed to converge or produced significant errors.

Sparse Text Data

The framework was applied to Bag-of-Words (BOW) and TF-IDF representations from the 20 Newsgroups and NeurIPS datasets. These datasets are characterized by high dimensionality (exceeding 12,000 dimensions) and extreme sparsity. JUMP demonstrated a significant advantage here: on the NeurIPS dataset, JUMP achieved a Wasserstein-1 distance for sparsity of 3.18, compared to 693.21 for DDPM—an improvement of two orders of magnitude.

Natural Images

To test its versatility, JUMP was evaluated on the CIFAR-10 benchmark. While JUMP is specialized for sparse data, it remained competitive with highly optimized Gaussian models, achieving an Inception Score (IS) of 9.04 (compared to 9.46 for DDPM).

Discussion and Conclusion

The "Learning to Jump" framework represents a significant step forward for generative modeling in domains where Gaussian assumptions do not hold. By treating zero as an absorbing state, JUMP naturally preserves sparsity, making it ideal for text and categorical data. Furthermore, its Poisson-based architecture provides inherent robustness to heavy-tailed distributions.

While JUMP currently faces challenges regarding sampling speed—requiring multiple iterative passes similar to standard diffusion—it offers a robust alternative for non-negative data. Future research into acceleration techniques and architectural optimizations for high-dimensional continuous data will likely further bridge the gap between JUMP and traditional diffusion models, making it a versatile tool for a wider range of scientific and generative AI applications.

Summary updated

Abstract

Learning to denoise has emerged as a prominent paradigm to design state-of-the-art deep generative models for natural images. How to use it to model the distributions of both continuous real-valued data and categorical data has been well studied in recently proposed diffusion models. However, it is found in this paper to have limited ability in modeling some other types of data, such as count and non-negative continuous data, that are often highly sparse, skewed, heavy-tailed, and/or overdispersed. To this end, we propose learning to jump as a general recipe for generative modeling of various types of data. Using a forward count thinning process to construct learning objectives to train a deep neural network, it employs a reverse count thickening process to iteratively refine its generation through that network. We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better. For example, learning to jump is recommended when the training data is non-negative and exhibits strong sparsity, skewness, heavy-tailedness, and/or heterogeneity.
BibTeX
@inproceedings{chen2023learning,
  author = {Tianqi Chen and Mingyuan Zhou},
  title = {{Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=Uj5AVsHkoX},
}
Figure