Asymptotic analysis of the jittering kernel density estimator

Thomas Nagler

Abstract

Jittering estimators are nonparametric function estimators for mixed data. They extend arbitrary estimators from the continuous setting by adding random noise to discrete variables. We give an in-depth analysis of the jittering kernel density estimator, which reveals several appealing properties. The estimator is strongly consistent, asymptotically normal, and unbiased for discrete variables. It converges at minimax-optimal rates, which are established as a by-product of our analysis. To understand the effect of adding noise, we further study its asymptotic efficiency and finite sample bias in the univariate discrete case. Simulations show that the estimator is competitive on finite samples. The analysis suggests that similar properties can be expected for other jittering estimators.

Introduction

In nonparametric statistics, Kernel Density Estimation (KDE) is a fundamental tool for modeling data distributions without the constraints of parametric assumptions. While standard KDE is highly effective for continuous variables, modern datasets frequently contain mixed data types—combining continuous measurements (e.g., height) with discrete categories (e.g., number of children). Applying standard KDE to these mixed datasets is non-trivial and often requires specialized, complex estimators.

To circumvent this complexity, practitioners often resort to "jittering": adding a small amount of random noise to discrete variables to treat them as continuous. Despite its widespread use, jittering has historically been dismissed as a crude heuristic that sacrifices statistical efficiency. This paper provides the rigorous mathematical framework to demonstrate that the Jittering Kernel Density Estimator (JKDE) is not just a convenient shortcut, but a theoretically sound, minimax-optimal method for mixed-data density estimation.

The Jittering Kernel Density Estimator (JKDE)

The JKDE approach transforms a mixed-data problem into a continuous one through a three-step process:

  1. Noise Addition: For every discrete observation $Z_i$, a random noise term $E_i$ is added. This noise is drawn from a specific class of distributions ($\mathcal{E}_{\gamma_1, \gamma_2}$) that are "flat" around zero and have a strictly limited range.
  2. Standard KDE Application: The classical Parzen-Rosenblatt window estimator is applied to the resulting "continuous" data.
  3. Bandwidth Selection: The estimator utilizes smoothing parameters (bandwidths) for both the discrete and continuous components to control the trade-off between bias and variance.

Theoretical Insights

The Vanishing Derivative Trick

A central contribution of this analysis is the discovery of the "vanishing derivative" effect. By selecting a specific type of noise, the density of the jittered data becomes constant over small intervals. This causes the derivatives of the density with respect to the discrete variables to vanish. In the context of KDE, this effectively eliminates the bias that typically arises when smoothing discrete data, allowing the estimator to focus on the continuous components.

Minimax Optimality and Efficiency

The paper establishes the first-ever minimax-optimal convergence rates for mixed-data density estimation. The JKDE achieves a convergence rate of $n^{-\beta/(2\beta+q)}$, where $\beta$ represents the smoothness of the continuous part and $q$ is the number of continuous variables. Crucially, the presence of discrete variables does not slow down this rate, meaning there is no theoretical "penalty" for adding discrete complexity or recoding categorical variables into multiple binary indicators.

Furthermore, the analysis addresses concerns regarding efficiency loss. While adding noise does introduce some variance, the loss is often negligible. Under specific conditions—such as using uniform noise and a uniform kernel—the JKDE becomes exactly as efficient as a simple frequency count estimator.

Empirical Validation

To validate these theoretical findings, the author conducted simulations comparing the JKDE against the Li-Racine estimator, the current "gold standard" for mixed-data nonparametric estimation. The simulations utilized vectors $(Z, X)$ where $Z$ followed a Binomial distribution and $X$ followed a Standard Normal distribution.

The results demonstrated that the JKDE is highly competitive. In scenarios with a high number of discrete categories (e.g., $m=15$), the jittering approach actually outperformed the Li-Racine estimator in terms of accuracy. The JKDE was also proven to be strongly consistent and asymptotically normal across these tests.

Limitations and Future Directions

While the JKDE shows great promise, the paper identifies areas for further refinement:

  • Small Sample Sensitivity: On very small samples, the theoretical preference for "large" discrete bandwidths can lead to over-smoothing. Practical applications currently rely on likelihood cross-validation, which is computationally demanding.
  • Data-Dependent Noise: The current model assumes noise is independent of the data. Future research into data-dependent noise could potentially enhance finite-sample performance.
  • Broader Applications: The appealing properties of the JKDE likely extend to other jittered models, such as Nadaraya-Watson regression, providing a fertile ground for future formal proofs.

Conclusion

By providing a rigorous asymptotic analysis, this paper elevates jittering from a practitioner's "hack" to a mathematically justified statistical strategy. The JKDE offers a simple, efficient, and minimax-optimal path for density estimation in the increasingly common world of mixed-type data.

Summary updated

Abstract

Jittering estimators are nonparametric function estimators for mixed data. They extend arbitrary estimators from the continuous setting by adding random noise to discrete variables. We give an in-depth analysis of the jittering kernel density estimator, which reveals several appealing properties. The estimator is strongly consistent, asymptotically normal, and unbiased for discrete variables. It converges at minimax-optimal rates, which are established as a by-product of our analysis. To understand the effect of adding noise, we further study its asymptotic efficiency and finite sample bias in the univariate discrete case. Simulations show that the estimator is competitive on finite samples. The analysis suggests that similar properties can be expected for other jittering estimators.
BibTeX
@inproceedings{nagler2018asymptotic,
  author = {Thomas Nagler},
  title = {{Asymptotic analysis of the jittering kernel density estimator}},
  booktitle = {Mathematical},
  year = {2018},
  doi = {10.3103/s1066530718010027},
}
Figure