Abstract
The infinite mixture of normals model has become a popular method for density estimation problems. This paper proposes an alternative hierarchical model that leads to hyperparameters that can be interpreted as the location, scale and smoothness of the density. The priors on other parts of the model have little effect on the density estimates and can be given default choices. Automatic Bayesian density estimation can be implemented by using uninformative priors for location and scale and default priors for the smoothness. The performance of these methods for density estimation are compared to previously proposed default priors for four data sets.
Introduction
Density estimation—the task of estimating the underlying probability distribution of a set of data points—is a cornerstone of statistics. Bayesian nonparametric methods, specifically Dirichlet Process Mixture Models (DPMMs), are powerful because they can adapt to almost any shape by treating the density as an infinite sum of simpler "atoms," usually Normal distributions.
However, these models are notoriously difficult to tune. Researchers must choose "priors" (initial beliefs) for various parameters, and standard choices often involve hyperparameters that lack clear physical meaning. If the priors are poorly chosen, the model might "over-smooth" the data (missing real peaks) or "under-smooth" it (creating fake peaks out of noise). This paper seeks a "default" prior that works well out-of-the-box without requiring expert tuning.
Methods
The author proposes a hierarchical infinite mixture of normals model with two main variants:
- Common Component Variance (CCV): All normal distributions in the mixture share the same variance.
- Different Component Variance (DCV): Each component can have its own variance, allowing the model to capture distributions with varying levels of "spread" in different regions.
The Key Innovation: Reparameterization
Instead of placing priors on the raw variance of the components, the author reparameterizes the model using a parameter $a$ (where $0 < a < 1$). This parameter acts as a smoothness parameter. When $a$ is close to 1, the model favors a smooth, unimodal shape; when $a$ is close to 0, the model allows for many sharp, distinct modes.
By placing a Beta prior on $a$, the user can express a general preference for smoothness while letting the data decide the final complexity. The model also uses uninformative (Jeffreys') priors for the overall location and scale, making the process largely automatic.
Key Insights and Ideas
- Smoothness is the "Master Knob": The paper demonstrates that the number of modes in the estimated density is much more sensitive to the smoothness parameter $a$ than to the mass parameter $M$ of the Dirichlet process. This significantly simplifies the tuning process.
- Intuitive Priors: By linking hyperparameters to location, scale, and smoothness, the author moves away from abstract mathematical tuning toward parameters that a researcher can actually visualize.
- Robustness to the Mixing Process: Testing the model using both the standard Dirichlet process and the more general Normalized Generalized Gamma process yielded remarkably similar results, suggesting that the structure of the hierarchy is more important than the specific type of infinite process used.
Key Results
The proposed CCV and DCV models were compared against two other popular "default" schemes: Richardson and Green (RG) and Ishwaran and James (IJ) across four classic benchmark datasets (Galaxy, Acidity, Enzyme, and Sodium Lithium).
- Predictive Performance: Using log predictive scores, the CCV and DCV models outperformed both RG and IJ across all four datasets.
- Flexibility: The DCV model was particularly effective for the "Enzyme" dataset, where the distribution has two modes with very different widths—a scenario where the simpler CCV model struggled slightly.
- Modality Capture: The model correctly identified the multi-modal nature of the Galaxy data, providing a more nuanced fit than previous "automatic" methods that tended to over-smooth it into a single peak.
Limitations and Future Work
While the paper focuses on observable data, these default priors could be even more valuable for "unobservable" quantities like random effects in complex models. Future work could allow the centering distribution to be skewed or heavy-tailed to better handle highly asymmetric data, or extend the framework to allow for explicit modeling of skewness and kurtosis.
Summary updated