Abstract
The benefits of the wavelet approach for density estimation are well established, particularly when the target density is irregular or exhibits heterogeneous smoothness. However, a persistent challenge in wavelet density estimation is that the resulting estimates are typically not "bona fide" densities—they may take negative values or fail to integrate to unity. This paper builds upon the "shape-preserving" wavelet density estimator introduced by Aya-Moreno et al. (2018), which utilizes a square-root transformation to ensure non-negativity. We deliver a fully data-driven version of this estimator where all critical parameters, such as the resolution level and thresholding specifications, are selected by optimizing an original leave-one-out version of the Hellinger-Bhattacharyya criterion. We establish the theoretical optimality of this procedure and demonstrate its strong practical performance through simulations. Additionally, we propose a novel "jackknife thresholding" scheme that proves superior to classical thresholding options.
Introduction
Density estimation—the process of reconstructing an underlying probability distribution from a set of observed data points—is a fundamental task in statistics. Wavelet-based methods have gained popularity in this field due to their ability to capture local features, such as sharp peaks or sudden changes in smoothness, which often baffle traditional global estimators.
Despite their flexibility, standard wavelet density estimators often produce non-bona fide densities. Because wavelets can oscillate, the resulting estimate $\hat{f}$ may yield negative values in regions with low data density. Furthermore, ensuring the estimate integrates to exactly one often requires ad-hoc post-processing. A robust solution to this is the "square-root trick": estimating the square root of the density, $g = \sqrt{f}$, and then squaring the result. Since any real number squared is non-negative, the final estimate $\hat{f} = \hat{g}^2$ is guaranteed to be a valid density. However, this approach introduces a new challenge: how to automatically select the resolution level ($J$) and the thresholding parameter ($\kappa$) without manual intervention.
Methodology
The Hellinger-Bhattacharyya Criterion
This paper introduces the Hellinger-Bhattacharyya cross-validation (HBCV) framework to automate parameter selection. The choice of the Hellinger distance is mathematically motivated; because the estimation process targets the square root of the density, the Hellinger distance—which is defined in terms of the difference between square roots—is the most natural metric for assessing error.
To select the optimal resolution level $J$, we maximize the Bhattacharyya affinity coefficient, a measure of similarity between two distributions. Since the true underlying density is unknown in real-world scenarios, we employ a consistent leave-one-out estimator based on nearest-neighbor distances. This allows the algorithm to "self-tune" based solely on the provided data.
Jackknife Thresholding
Wavelet coefficients are typically "thresholded" to separate signal from noise. Traditional methods often use a "universal threshold," which may be too aggressive or too lenient depending on the local variance of the data. We propose a "jackknife thresholding" scheme that uses leave-one-out pseudo-values to estimate the specific variance of each individual wavelet coefficient. This results in a threshold that adapts to the specific noise characteristics of the dataset, providing a more nuanced reconstruction of the density's features.
Experimental Design
Synthetic Simulations
The proposed estimator was tested against four distinct 2D synthetic density mixtures designed to challenge different aspects of the algorithm:
- 2D Comb: Features multiple sharp, narrow peaks.
- Pyramids Mix: Tests the ability to handle non-smooth, angular densities.
- 2D Gaussian Mix 1: Features high locality with varying spreads.
- 2D Gaussian Mix 2: Features anisotropic (elongated) covariances.
Real-World Application
We applied the estimator to the classic Old Faithful geyser dataset. This bivariate dataset, consisting of eruption durations and waiting times between eruptions, serves as a benchmark for evaluating how well the estimator captures the distinct clusters and correlations present in real-world phenomena.
Key Results and Discussion
Asymptotic Optimality
We provide mathematical proof that the HBCV data-driven selection of the resolution level $J$ is asymptotically optimal. In practice, the procedure tends to be slightly conservative, often selecting a resolution level one unit lower than the theoretical "oracle" optimum. This conservatism acts as a safeguard against over-fitting, particularly in smaller sample sizes.
Performance vs. Kernel Density Estimation (KDE)
While Kernel Density Estimation (KDE) remains a strong performer for very small samples ($n < 500$), the shape-preserving wavelet estimator demonstrates clear superiority as the sample size increases ($n \geq 1000$). This is especially evident in densities with complex local structures where KDE tends to oversmooth. Furthermore, the wavelet approach is highly efficient; for a sample of $n=6000$, the estimator achieved higher accuracy than KDE while utilizing fewer than 500 non-zero coefficients, demonstrating significant sparsity.
Thresholding Efficacy
In head-to-head comparisons, the jackknife thresholding scheme consistently outperformed traditional universal or level-dependent thresholding. By adapting to the local variance, the jackknife method preserved essential structural details that other methods often smoothed away as noise.
Conclusion and Future Directions
The HBCV framework provides a robust, fully automated path for shape-preserving density estimation. While the current implementation uses a tensor-product construction for multivariate data, future work could improve performance on highly anisotropic data by integrating more advanced multiscale structures such as curvelets or shearlets. Additionally, while the jackknife thresholding shows empirical superiority, further theoretical research is required to fully characterize its formal statistical properties compared to classical thresholding strategies.
Summary updated