Latent feature regression for multivariate count data.

Arto Klami, Abhishek Tripathi, Johannes Sirola, Lauri Väre, Frédéric RoullandAISTATS 2015 · OpenReview

Abstract

Predicting multiple related count variables is a fundamental challenge in data science, particularly when the data exhibits complex dispersion patterns. We present a novel Bayesian framework, Latent Feature Regression (LFR), which combines Negative Binomial regression with a Dirichlet-multinomial observation layer. This architecture allows the model to learn hidden processes generating counts while capturing correlations between outputs. Our results show that LFR effectively handles both overdispersed and underdispersed data, significantly outperforming traditional methods on synthetic benchmarks and real-world urban traffic datasets.

Introduction

In many real-world applications, such as urban planning or ecology, we need to perform multivariate regression on count data. For instance, predicting the number of passengers boarding buses at different stops simultaneously requires a model that understands both the influence of local geography and the correlations between stops. While Gaussian processes are effective for continuous data, count data is often "messy." It frequently exhibits overdispersion (high variance, common in rare bursts of activity) or underdispersion (low variance, typical in regulated flows).

Standard approaches often rely on the Poisson distribution, which is limited by the strict assumption that variance equals the mean. Negative Binomial (NB) models can handle overdispersion but struggle with underdispersion. Furthermore, existing multivariate tools like Vector Generalized Linear Models (VGLM) often lack robustness or fail to provide a full probabilistic representation of correlations.

The Latent Feature Regression Framework

The proposed LFR model addresses these issues through a two-stage generative process inspired by topic modeling:

  1. The Latent Layer: The model maps input covariates ($x$) to a set of $K$ latent count-valued features ($z$). These features are modeled using Negative Binomial regression, allowing the system to learn "hidden processes" that drive the observed counts.
  2. The Observation Layer: These latent counts are distributed across the $L$ observed output variables via a Dirichlet-multinomial distribution.
Probabilistic graphical model of the Latent Feature Regression (LFR) framework. The model maps covariates to latent features (z) which then generate observed counts (y).

A key advantage of this structure is its scalability. Unlike naive methods that require a separate model for every output variable, the complexity of LFR is independent of the number of output dimensions ($L$). Inference is made computationally feasible through a Gibbs sampler utilizing Pólya-Gamma augmentation for logistic mapping and a Chinese Restaurant Process representation for NB parameters, enabling efficient closed-form updates.

Modeling Dispersion and Correlations

The LFR model cleverly frames multivariate regression as a generalization of a topic model. While standard topic models like Latent Dirichlet Allocation (LDA) focus on relative frequencies, LFR predicts actual total counts.

One of the most significant technical insights is the model's ability to handle underdispersion. By "splitting" the generation process across multiple latent topics and then summing them back up through the Dirichlet-multinomial layer, the model naturally reduces variance. This allows it to fit underdispersed data—a feat that traditional Negative Binomial models cannot achieve.

Average likelihood vs. number of latent features (K) for overdispersed and underdispersed synthetic data. LFR (black) demonstrates superior performance and stability compared to the LGNB baseline (green), especially in the underdispersed case.

Experimental Evaluation

The model was evaluated on both artificial and real-world datasets:

  • Synthetic Benchmarks: On underdispersed data generated via Conway-Maxwell-Poisson distributions, LFR with a large $K$ vastly outperformed traditional NB models, which are mathematically incapable of modeling variance lower than the mean.
  • Smart Traffic Application: The model was applied to a dataset of 21 days of bus traffic in a major city, predicting passenger volume across 37 routes and 603 stops. Input features included 18 categories of nearby venues from Foursquare (e.g., restaurants, shops, schools). LFR successfully captured 35% of the variance in passenger counts using only static neighborhood characteristics, significantly outperforming independent LGNB models and regularized GLMs in terms of predictive log-likelihood.

Conclusion and Future Work

Latent Feature Regression offers a robust, scalable solution for multivariate count data. By leveraging latent features, it captures complex correlations and dispersion patterns that stymie traditional models. Future research will focus on improving computational scaling for extremely high counts, which currently scale linearly with the sum of counts, and incorporating zero-inflation to better handle sparse datasets where many observed counts are zero.

Summary updated

Abstract

We consider the problem of regression on multivariate count data and present a Gibbs sampler for a latent feature regression model suitable for both under- and overdispersed response variables. The model learns count-valued latent features conditional on arbitrary covariates, modeling them as negative binomial variables, and maps them into the dependent count-valued observations using a Dirichlet-multinomial distribution. From another viewpoint, the model can be seen as a generalization of a specific topic model for scenarios where we are interested in generating the actual counts of observations and not just their relative frequencies and co-occurrences. The model is demonstrated on a smart traffic application where the task is to predict public transportation volume for unknown locations based on a characterization of the close-by services and venues.
BibTeX
@inproceedings{klami2015latent,
  author = {Arto Klami and Abhishek Tripathi and Johannes Sirola and Lauri Väre and Frédéric Roulland},
  title = {{Latent feature regression for multivariate count data.}},
  booktitle = {AISTATS},
  year = {2015},
  url = {https://openreview.net/forum?id=DnZom0qOV41s},
}
Figure