Hellinger Distance and Bayesian Non-Parametrics: Hierarchical Models for Robust and Efficient Bayesian Inference

Yuefeng Wu, Giles HookerarXiv 2013 · DOI · arXiv

Abstract

This paper introduces a novel hierarchical framework that integrates Hellinger distance methods into Bayesian analysis to achieve both robustness to outliers and statistical efficiency. By placing a non-parametric density layer between the data and the parametric model, we create a "Hellinger posterior" that naturally downweights observations inconsistent with the assumed parametric family. We demonstrate that this approach maintains the precision of traditional Bayesian methods when the model is correct while providing significant protection against contamination, as evidenced by simulations and an analysis of equine parasite data.

Introduction: The Efficiency-Robustness Trade-off

In statistical modeling, practitioners often face a fundamental conflict between efficiency and robustness. Efficiency refers to the precision of estimates when the assumed model (e.g., a Normal distribution) is correct. Robustness refers to the stability of those estimates when the model is slightly misspecified or the data contains outliers.

Standard Bayesian methods are highly efficient but notoriously sensitive to outliers. While heavy-tailed distributions like the Student’s t-distribution can offer some protection, they are often specific to certain parameters and lack a general framework. Frequentist statistics has long utilized Minimum Hellinger Distance (MHD) estimation to bridge this gap, but a cohesive Bayesian equivalent has been elusive. This research proposes a hierarchical Bayesian framework that uses Hellinger distance as a disparity measure to protect the model from data that deviates from the parametric assumption.

Methodology

The proposed framework utilizes a two-pronged approach to integrate Hellinger distance into the Bayesian workflow:

1. One-Step Methods

The authors first demonstrate that a Bayesian non-parametric (BNP) density estimate, such as a Dirichlet Process Mixture, can replace the traditional frequentist kernel density estimate. This BNP estimate is then projected onto a parametric family using the Hellinger distance, providing a robust point estimate.

2. The Hierarchical Hellinger Model

The primary contribution of this work is a hierarchical model where a non-parametric density ($g$) is placed between the data and the parameters ($\theta$). The likelihood is modified by the exponential of twice the Hellinger distance between the candidate non-parametric density and the parametric family:

$$L(\theta) \propto \exp(-2nH^2(g, f_\theta))$$

In this structure, the non-parametric layer absorbs the "noise" or outliers in the raw data. The Hellinger distance ensures that the parameters of interest ($\theta$) are primarily influenced by the portion of the data that aligns with the assumed parametric shape ($f_\theta$). This creates a "bridge" that allows the model to be parametric in its goals but non-parametric in its handling of data anomalies.

Experimental Evaluation

Simulated Gaussian Data

To test the model's resilience, the authors generated 1,000 datasets ($n=20$) from a $N(5, 1)$ distribution. These sets were contaminated with outliers at extreme locations (-3, -5, and -10).

  • Standard Bayesian Posterior: Showed massive sensitivity. With 5 outliers at -10, the bias reached -2.092.
  • Hierarchical Hellinger Method: Maintained a bias of only 0.018 under the same conditions.

This result confirms that the Hellinger framework effectively ignores extreme outliers that would otherwise derail a standard analysis.

Equine Parasite Data

The method was applied to a real-world dataset from a parasite control study in Denmark, tracking "Strongyle" egg counts in horses before and after treatment with Pyrantol. This data is naturally over-dispersed and contains outliers. The Hellinger method provided stable estimates of the mean and variance of the log-odds of parasite survival, successfully identifying specific horse reactions as outliers that would have skewed drug efficacy results in a standard model.

Discussion and Insights

The research proves mathematically that if the data is "clean" and follows the parametric model, the Hellinger method achieves asymptotic efficiency—meaning there is no "penalty" for using this robust framework. It provides automatic outlier rejection based on the geometry of probability distributions rather than manual data cleaning.

Limitations and Future Work

While powerful, the approach is computationally intensive, requiring MCMC sampling (Metropolis-within-Gibbs) for the non-parametric layer. Future research will explore other disparity measures, such as the negative exponential disparity, and extend the framework from i.i.d. data to more complex regression and random-effects models.

Summary updated

Abstract

This paper introduces a hierarchical framework to incorporate Hellinger distance methods into Bayesian analysis. We propose to modify a prior over non-parametric densities with the exponential of twice the Hellinger distance between a candidate and a parametric density. By incorporating a prior over the parameters of the second density, we arrive at a hierarchical model in which a non-parametric model is placed between parameters and the data. The parameters of the family can then be estimated as hyperparameters in the model. In frequentist estimation, minimizing the Hellinger distance between a kernel density estimate and a parametric family has been shown to produce estimators that are both robust to outliers and statistically efficient when the parametric model is correct. In this paper, we demonstrate that the same results are applicable when a non-parametric Bayes density estimate replaces the kernel density estimate. We then demonstrate that robustness and efficiency also hold for the proposed hierarchical model. The finite-sample behavior of the resulting estimates is investigated by simulation and on real world data.
BibTeX
@misc{wu2013hellinger,
  author = {Yuefeng Wu and Giles Hooker},
  title = {{Hellinger Distance and Bayesian Non-Parametrics: Hierarchical Models for Robust and Efficient Bayesian Inference}},
  year = {2013},
  eprint = {1309.6906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1309.6906},
}

Loading related content...

Figure