Modelling and Computation Using NCoRM Mixtures for Density Regression

Jim E. Griffin, Fabrizio LeisenBayesian 2017 · DOI

TL;DR

This paper introduces a new class of flexible, non-parametric regression models called Normalized Compound Random Measure (NCoRM) mixture models. They develop a novel computational method, a pseudo-marginal Metropolis-Hastings sampler, to perform inference for these models, enabling their application to density regression problems.

What Problem Does This Paper Address?

The paper tackles the challenge of Bayesian non-parametric inference for distributions that depend on continuous covariates. Existing methods often struggle with flexibility and computational tractability when dealing with complex dependencies. The authors aim to provide a more general and flexible framework for density regression by extending the concept of compound random measures (CoRMs) to infinite collections of distributions, allowing for richer dependency structures.

Methods

The core of the paper lies in two main contributions:

  • NCoRM Mixture Models: They define a new class of mixture models where the mixing measure is a normalized compound random measure. This allows for flexible modeling of density regression where the mixture components' weights can depend on the regressor. The model is a generalization of existing approaches and offers advantages like not requiring ordered weights.
  • Pseudo-Marginal Metropolis-Hastings Sampler: To overcome the computational intractability of exact posterior inference, they develop a novel MCMC algorithm. This algorithm uses a pseudo-marginal approach, replacing an intractable component of the target distribution (the Laplace transform of a CoRM) with an unbiased estimator. This allows for efficient posterior inference in a wider range of NCoRM models.

Datasets

The paper uses two types of datasets:

  • Simulated Data Sets: Three sets of simulated data are generated to evaluate the performance of the proposed NCoRM models against existing methods (probit stick-breaking and dependent Dirichlet process). These datasets are designed to represent different scenarios, including varying levels of separation between mixture components, non-linear means, and heteroscedasticity.
  • Real-World Data:
    • Hematological Data: Analyzed in Example 1, this data comes from a dose-escalation study of white blood cell counts over time for patients receiving different treatments.
  • Motorcycle Accident Data: Used in Example 2, this data consists of head accelerations measured at different times after impact, available in the R package MASS.

Key Insights and Ideas

  • Flexibility through NCoRM: The NCoRM framework provides a powerful way to model complex dependencies in density regression by allowing mixture weights to vary with regressors. This moves beyond simpler models where weights are constant or depend only on discrete categories.
  • Bridging the Computational Gap: The pseudo-marginal MCMC approach is a significant methodological contribution. It allows for the application of sophisticated Bayesian non-parametric models that were previously computationally infeasible due to intractable integrals.
  • Unbiased Estimation of Laplace Functionals: The development of a general approach for unbiasedly estimating Laplace functionals of compound random measures is a key enabler for the MCMC method. This technique is broadly applicable beyond the specific NCoRM models discussed.
  • Performance Gains: The NCoRM models, coupled with the new sampler, demonstrate superior out-of-sample predictive performance compared to existing methods in several simulated scenarios, particularly when the mean of the responses does not vary smoothly.

Key Results

  • The proposed NCoRM mixture models are shown to be effective for density regression, capturing complex relationships between regressors and conditional densities.
  • The pseudo-marginal Metropolis-Hastings sampler is demonstrated to be efficient and applicable to NCoRM mixture models, enabling posterior inference.
  • In simulation studies, the NCoRM approach often outperformed probit stick-breaking and dependent Dirichlet process models in terms of out-of-sample predictive performance, especially in scenarios with non-smooth response means.
  • The analysis of real-world hematological and motorcycle accident data showcases the practical utility of the NCoRM models in modeling complex data structures.

Limitations and Future Work

  • Computational Cost: While the pseudo-marginal sampler is efficient, MCMC methods for non-parametric models can still be computationally intensive, especially with complex models or large datasets.
  • Model Complexity: The flexibility of NCoRM models, while an advantage, can also lead to overfitting if not carefully managed.
  • Future Work: The authors suggest that the NCoRM approach can be generalized to allow jump locations to depend on regressors, and the MCMC method can be extended. They also highlight the potential to incorporate other types of dependence structures (e.g., time series, spatial, hierarchical models) and explore different classes of Lévy processes. The development of more efficient approximations for tail mass integrals of Lévy processes is also mentioned as an area for further research.

Summary updated

Abstract

Normalized compound random measures are flexible nonparametric priors for related distributions. We consider building general nonparametric regression models using normalized compound random measure mixture models. Posterior inference is made using a novel pseudo-marginal Metropolis-Hastings sampler for normalized compound random measure mixture models. The algorithm makes use of a new general approach to the unbiased estimation of Laplace functionals of compound random measures (which includes completely random measures as a special case). The approach is illustrated on problems of density regression.
BibTeX
@inproceedings{griffin2017modelling,
  author = {Jim E. Griffin and Fabrizio Leisen},
  title = {{Modelling and Computation Using NCoRM Mixtures for Density Regression}},
  booktitle = {Bayesian},
  year = {2017},
  doi = {10.1214/17-ba1072},
}

Loading related content...

Figure