Hellinger-Bhattacharyya cross-validation for shape-preserving multivariate wavelet thresholding

Carlos Aya-Moreno, Gery Geenens, Spiridon PenevarXiv 2022 · DOI · arXiv

Abstract

The benefits of the wavelet approach for density estimation are well established, particularly when the target density is irregular or exhibits heterogeneous smoothness. However, a persistent challenge in wavelet density estimation is that the resulting estimates are typically not "bona fide" densities—they may take negative values or fail to integrate to unity. This paper builds upon the "shape-preserving" wavelet density estimator introduced by Aya-Moreno et al. (2018), which utilizes a square-root transformation to ensure non-negativity. We deliver a fully data-driven version of this estimator where all critical parameters, such as the resolution level and thresholding specifications, are selected by optimizing an original leave-one-out version of the Hellinger-Bhattacharyya criterion. We establish the theoretical optimality of this procedure and demonstrate its strong practical performance through simulations. Additionally, we propose a novel "jackknife thresholding" scheme that proves superior to classical thresholding options.

Introduction

Density estimation—the process of reconstructing an underlying probability distribution from a set of observed data points—is a fundamental task in statistics. Wavelet-based methods have gained popularity in this field due to their ability to capture local features, such as sharp peaks or sudden changes in smoothness, which often baffle traditional global estimators.

Despite their flexibility, standard wavelet density estimators often produce non-bona fide densities. Because wavelets can oscillate, the resulting estimate $\hat{f}$ may yield negative values in regions with low data density. Furthermore, ensuring the estimate integrates to exactly one often requires ad-hoc post-processing. A robust solution to this is the "square-root trick": estimating the square root of the density, $g = \sqrt{f}$, and then squaring the result. Since any real number squared is non-negative, the final estimate $\hat{f} = \hat{g}^2$ is guaranteed to be a valid density. However, this approach introduces a new challenge: how to automatically select the resolution level ($J$) and the thresholding parameter ($\kappa$) without manual intervention.

Methodology

The Hellinger-Bhattacharyya Criterion

This paper introduces the Hellinger-Bhattacharyya cross-validation (HBCV) framework to automate parameter selection. The choice of the Hellinger distance is mathematically motivated; because the estimation process targets the square root of the density, the Hellinger distance—which is defined in terms of the difference between square roots—is the most natural metric for assessing error.

To select the optimal resolution level $J$, we maximize the Bhattacharyya affinity coefficient, a measure of similarity between two distributions. Since the true underlying density is unknown in real-world scenarios, we employ a consistent leave-one-out estimator based on nearest-neighbor distances. This allows the algorithm to "self-tune" based solely on the provided data.

Jackknife Thresholding

Wavelet coefficients are typically "thresholded" to separate signal from noise. Traditional methods often use a "universal threshold," which may be too aggressive or too lenient depending on the local variance of the data. We propose a "jackknife thresholding" scheme that uses leave-one-out pseudo-values to estimate the specific variance of each individual wavelet coefficient. This results in a threshold that adapts to the specific noise characteristics of the dataset, providing a more nuanced reconstruction of the density's features.

Experimental Design

Synthetic Simulations

The proposed estimator was tested against four distinct 2D synthetic density mixtures designed to challenge different aspects of the algorithm:

  • 2D Comb: Features multiple sharp, narrow peaks.
  • Pyramids Mix: Tests the ability to handle non-smooth, angular densities.
  • 2D Gaussian Mix 1: Features high locality with varying spreads.
  • 2D Gaussian Mix 2: Features anisotropic (elongated) covariances.

Real-World Application

We applied the estimator to the classic Old Faithful geyser dataset. This bivariate dataset, consisting of eruption durations and waiting times between eruptions, serves as a benchmark for evaluating how well the estimator captures the distinct clusters and correlations present in real-world phenomena.

Key Results and Discussion

Asymptotic Optimality

We provide mathematical proof that the HBCV data-driven selection of the resolution level $J$ is asymptotically optimal. In practice, the procedure tends to be slightly conservative, often selecting a resolution level one unit lower than the theoretical "oracle" optimum. This conservatism acts as a safeguard against over-fitting, particularly in smaller sample sizes.

Performance vs. Kernel Density Estimation (KDE)

While Kernel Density Estimation (KDE) remains a strong performer for very small samples ($n < 500$), the shape-preserving wavelet estimator demonstrates clear superiority as the sample size increases ($n \geq 1000$). This is especially evident in densities with complex local structures where KDE tends to oversmooth. Furthermore, the wavelet approach is highly efficient; for a sample of $n=6000$, the estimator achieved higher accuracy than KDE while utilizing fewer than 500 non-zero coefficients, demonstrating significant sparsity.

Thresholding Efficacy

In head-to-head comparisons, the jackknife thresholding scheme consistently outperformed traditional universal or level-dependent thresholding. By adapting to the local variance, the jackknife method preserved essential structural details that other methods often smoothed away as noise.

Conclusion and Future Directions

The HBCV framework provides a robust, fully automated path for shape-preserving density estimation. While the current implementation uses a tensor-product construction for multivariate data, future work could improve performance on highly anisotropic data by integrating more advanced multiscale structures such as curvelets or shearlets. Additionally, while the jackknife thresholding shows empirical superiority, further theoretical research is required to fully characterize its formal statistical properties compared to classical thresholding strategies.

Summary updated

Abstract

The benefits of the wavelet approach for density estimation are well established in the literature, especially when the density to estimate is irregular or heterogeneous in smoothness. However, wavelet density estimates are typically not bona fide densities. In Aya-Moreno et al (2018), a `shape-preserving' wavelet density estimator was introduced, including as main step the estimation of the square-root of the density. A natural concept involving square-root of densities is the Hellinger distance - or equivalently, the Bhattacharyya affinity coefficient. In this paper, we deliver a fully data-driven version of the above 'shape-preserving' wavelet density estimator, where all user-defined parameters, such as resolution level or thresholding specifications, are selected by optimising an original leave-one-out version of the Hellinger-Bhattacharyya criterion. The theoretical optimality of the proposed procedure is established, while simulations show the strong practical performance of the estimator. Within that framework, we also propose a novel but natural 'jackknife thresholding' scheme, which proves superior to other, more classical thresholding options.
BibTeX
@misc{ayamoreno2022hellinger,
  author = {Carlos Aya-Moreno and Gery Geenens and Spiridon Penev},
  title = {{Hellinger-Bhattacharyya cross-validation for shape-preserving multivariate wavelet thresholding}},
  year = {2022},
  eprint = {2204.13854},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.13854},
}
Figure