Abstract
Predicting multiple related count variables is a fundamental challenge in data science, particularly when the data exhibits complex dispersion patterns. We present a novel Bayesian framework, Latent Feature Regression (LFR), which combines Negative Binomial regression with a Dirichlet-multinomial observation layer. This architecture allows the model to learn hidden processes generating counts while capturing correlations between outputs. Our results show that LFR effectively handles both overdispersed and underdispersed data, significantly outperforming traditional methods on synthetic benchmarks and real-world urban traffic datasets.
Introduction
In many real-world applications, such as urban planning or ecology, we need to perform multivariate regression on count data. For instance, predicting the number of passengers boarding buses at different stops simultaneously requires a model that understands both the influence of local geography and the correlations between stops. While Gaussian processes are effective for continuous data, count data is often "messy." It frequently exhibits overdispersion (high variance, common in rare bursts of activity) or underdispersion (low variance, typical in regulated flows).
Standard approaches often rely on the Poisson distribution, which is limited by the strict assumption that variance equals the mean. Negative Binomial (NB) models can handle overdispersion but struggle with underdispersion. Furthermore, existing multivariate tools like Vector Generalized Linear Models (VGLM) often lack robustness or fail to provide a full probabilistic representation of correlations.
The Latent Feature Regression Framework
The proposed LFR model addresses these issues through a two-stage generative process inspired by topic modeling:
- The Latent Layer: The model maps input covariates ($x$) to a set of $K$ latent count-valued features ($z$). These features are modeled using Negative Binomial regression, allowing the system to learn "hidden processes" that drive the observed counts.
- The Observation Layer: These latent counts are distributed across the $L$ observed output variables via a Dirichlet-multinomial distribution.
A key advantage of this structure is its scalability. Unlike naive methods that require a separate model for every output variable, the complexity of LFR is independent of the number of output dimensions ($L$). Inference is made computationally feasible through a Gibbs sampler utilizing Pólya-Gamma augmentation for logistic mapping and a Chinese Restaurant Process representation for NB parameters, enabling efficient closed-form updates.
Modeling Dispersion and Correlations
The LFR model cleverly frames multivariate regression as a generalization of a topic model. While standard topic models like Latent Dirichlet Allocation (LDA) focus on relative frequencies, LFR predicts actual total counts.
One of the most significant technical insights is the model's ability to handle underdispersion. By "splitting" the generation process across multiple latent topics and then summing them back up through the Dirichlet-multinomial layer, the model naturally reduces variance. This allows it to fit underdispersed data—a feat that traditional Negative Binomial models cannot achieve.
Experimental Evaluation
The model was evaluated on both artificial and real-world datasets:
- Synthetic Benchmarks: On underdispersed data generated via Conway-Maxwell-Poisson distributions, LFR with a large $K$ vastly outperformed traditional NB models, which are mathematically incapable of modeling variance lower than the mean.
- Smart Traffic Application: The model was applied to a dataset of 21 days of bus traffic in a major city, predicting passenger volume across 37 routes and 603 stops. Input features included 18 categories of nearby venues from Foursquare (e.g., restaurants, shops, schools). LFR successfully captured 35% of the variance in passenger counts using only static neighborhood characteristics, significantly outperforming independent LGNB models and regularized GLMs in terms of predictive log-likelihood.
Conclusion and Future Work
Latent Feature Regression offers a robust, scalable solution for multivariate count data. By leveraging latent features, it captures complex correlations and dispersion patterns that stymie traditional models. Future research will focus on improving computational scaling for extremely high counts, which currently scale linearly with the sum of counts, and incorporating zero-inflation to better handle sparse datasets where many observed counts are zero.
Summary updated