TL;DR
Standard neural networks struggle to output calibrated uncertainties for discrete count data, either forcing continuous Gaussian approximations that predict fractional or negative counts, or relying on restrictive Poisson/Negative Binomial assumptions. The Deep Double Poisson Network (DDPN) solves this by parameterizing the flexible Double Poisson distribution, allowing independent modeling of mean and dispersion (handling under-, equi-, and over-dispersion). Paired with a tunable loss formulation ($\beta$-DDPN) and ensembling, DDPN delivers state-of-the-art predictive accuracy, well-calibrated discrete probabilities, and superior out-of-distribution (OOD) detection across tabular, image, point cloud, and text data.
What Problem Does This Paper Address?
Predicting discrete counts—such as crowd sizes, product inventory, or review ratings—requires quantifying observation noise (aleatoric uncertainty). Deep practitioners typically take one of two flawed paths:
- Gaussian heteroscedastic regression: Treats counts as continuous variables. This leads to clear pathologies: placing probability mass on non-integers, predicting impossible negative counts near zero, and outputting fractional credible intervals.
- Standard discrete networks: Predict parameters for Poisson or Negative Binomial distributions. However, Poisson assumes equi-dispersion (variance equals the mean), while Negative Binomial assumes over-dispersion (variance must be greater than or equal to the mean). Neither can handle under-dispersion (variance smaller than the mean), severely limiting their flexibility.
Meanwhile, classical Generalized Linear Models (GLMs) that use flexible count distributions cannot scale to complex inputs like images or text and restrict dispersion to be an explicit function of the mean.
Methods
The authors introduce Deep Double Poisson Networks (DDPN):
- Double Poisson Parameterization: A neural network backbone maps input $x$ to separate linear output layers for $\log \hat{\mu}$ (mean) and $\log \hat{\phi}$ (inverse dispersion) under the Double Poisson distribution, where $\mathbb{E}[Y] \approx \mu$ and $\text{Var}[Y] \approx \mu / \phi$. Because $\phi$ can be freely learned, the model naturally captures under-dispersion ($\phi > 1$), equi-dispersion ($\phi = 1$), and over-dispersion ($\phi < 1$).
- $\beta$-DDPN Objective: In heteroscedastic training, large predicted variance can drive the gradient of the mean toward zero, causing the network to explain away prediction errors as uncertainty. The authors introduce a stop-gradient weighting term $\lfloor \hat{\phi}_i^{-\beta} \rfloor$ with $\beta \in [0, 1]$. Setting $\beta = 0$ corresponds to standard Double Poisson negative log-likelihood (NLL), while $\beta = 1$ isolates the mean updates from $\phi$, providing a continuous dial to prioritize mean accuracy over calibration.
- Deep Ensembles: By training $M$ independently initialized DDPNs, the authors model epistemic uncertainty via uniform mixture distributions, obtaining exact closed-form ensemble mean and variance.
Datasets
The authors evaluate DDPN on a broad spectrum of modalities:
- Synthetic 1D Benchmarks: Non-linear functions exhibiting varying dispersion and under-dispersion regimes, as well as a Negative Binomial misspecification test.
- Tabular Datasets:
- Bikes: Hourly bike-share rental demand (high-frequency counts).
- Collision: UK Road Safety casualty counts (low-frequency, right-skewed).
- Complex Data Regimes:
- COCO-People (Images): Counting people in MS-COCO images using a Vision Transformer (ViT) backbone.
- Inventory (3D Point Clouds): Estimating retail shelf product counts using a CountNet3D backbone.
- Amazon Reviews (Text): Predicting discrete 1–5 star ratings from customer reviews using a DistilBERT backbone.
- OOD Benchmark: Verses from the King James Version (KJV) of the Bible evaluated on models trained on Amazon Reviews.
Key Insights and Ideas
- True Heteroscedastic Flexibility for Counts: By completely decoupling $\mu$ and $\phi$ within a discrete probability mass function, DDPN successfully models sharp, under-dispersed counts where standard Poisson and Negative Binomial DNNs fail.
- Misspecification Robustness: Even when synthetic data is drawn strictly from a Negative Binomial process, DDPN recovers the ground-truth distribution and outperforms a dedicated Negative Binomial network.
- Reversion to the Optimal Constant Solution under OOD: Under extreme distribution shift (evaluating biblical text on an Amazon review regressor), standard models output falsely confident predictions. Standard DDPN smoothly collapses into a diffuse, high-entropy distribution across all classes, indicating true unfamiliarity.
Key Results
- Tabular Performance: On Bikes, DDPN achieved the best NLL ($4.67$ single, $4.57$ ensemble) compared to Gaussian DNN ($5.00$) and Negative Binomial DNN ($4.72$). On Collision, $\beta_{1.0}$-DDPN obtained the top single-model MAE ($0.269$) and strong NLL ($0.707$).
- Complex Modalities:
- COCO-People: $\beta_{1.0}$-DDPN achieved the lowest MAE ($1.962$ single, $1.701$ ensemble), while standard DDPN scored the lowest single-model NLL ($2.251$).
- Inventory: DDPN tied for the best single-model MAE ($0.900$) and achieved the best single-model NLL ($1.555$).
- Out-of-Distribution Detection: In permutation tests comparing entropy on in-distribution reviews vs. out-of-distribution Bible verses, DDPN achieved an average mean difference of $\bar{\Delta} = 0.460$ ($p = 0.001$), outperforming Gaussian DNN ($\bar{\Delta} = 0.238$) and drastically outperforming Poisson ($\bar{\Delta} = 0.018$) and Negative Binomial DNNs ($\bar{\Delta} = 0.004$), which failed entirely to signal OOD shifts.
Limitations and Future Work
- Very High-Frequency Counts: The authors focused on low-to-moderate counts. For counts in the thousands or millions, the discrete pathologies of continuous Gaussians diminish, and Gaussian approximations may suffice.
- Moment Approximations: The relations $\mathbb{E}[Z] \approx \mu$ and $\text{Var}[Z] \approx \mu / \phi$ are established analytical approximations, but edge-case deviations from true Double Poisson moments warrant deeper theoretical study.
- Optimization Stability: DDPN exhibited convergence sensitivities during training with standard SGD or Adam, requiring AdamW with learning rate warmup/schedules to ensure stability.
- OOD Scope: Qualitative and quantitative OOD experiments were limited to text data (Amazon Reviews vs. KJV Bible) and should be extended to vision and 3D modalities.
Summary updated