Kernel Density Estimation Bias under Minimal Assumptions

Maciej SkorskiarXiv 2019 · OpenReview

TL;DR

This paper tackles the theoretical underpinnings of Kernel Density Estimation (KDE), a common method for approximating probability distributions. The authors show that to ensure KDE estimates are well-behaved (bounded), there's a crucial interplay between how quickly the kernel function decays and how the bandwidth matrix's eigenvalues shrink. They provide rigorous bias bounds under minimal assumptions, clarifying that sufficient kernel decay can simplify density assumptions often required in existing literature.

What Problem Does This Paper Address?

Kernel Density Estimation (KDE) is a fundamental tool for estimating an unknown probability density function from a set of data points. While widely used, the theoretical guarantees for its accuracy, particularly concerning the bias of the estimator, often rely on strong assumptions. These assumptions can include the kernel having compact support (meaning it's zero outside a bounded region) or the density function being very smooth. The paper highlights that existing literature sometimes invokes these assumptions without fully justifying them or overlooks subtle but critical points, such as the need for a global property (compactness) when applying local approximations like Taylor expansions. The core problem is to derive robust theoretical bounds on the bias of KDE that hold under more general and minimal conditions, specifically addressing the behavior of the kernel's decay and the structure of the bandwidth matrix.

Methods

The paper's approach is primarily theoretical and analytical. The authors delve into the mathematical formulation of KDE bias, which arises from the convolution of the kernel with the true density. They leverage Taylor expansions of the density function to express the bias.

Key methodological contributions include:

  • Analysis of Kernel Decay and Bandwidth Eigenvalues: The paper demonstrates a critical balance required between the decay rate of the kernel function (how quickly it approaches zero as its input grows large) and the magnitudes of the eigenvalues of the bandwidth matrix. They show that without sufficient kernel decay, KDE estimates can become unbounded, even if the density is smooth. This is a novel insight into the necessary conditions for KDE's well-posedness.
  • Rigorous Bias Bounds: The authors derive explicit, constant-bounded formulas for the bias of KDE. These bounds connect the kernel's decay, the norm and determinant of the bandwidth matrix, and the local smoothness of the density. This provides a more precise understanding of how these factors influence the estimation error.
  • Generalization of Taylor Expansion Application: They address the common practice of applying Taylor expansions globally. The paper shows that by carefully considering the kernel's decay, one can avoid the need for overly restrictive assumptions on the density's smoothness, which is a common workaround in existing literature.

Datasets

Not discussed in the paper. The paper is theoretical and focuses on deriving general mathematical bounds rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • The "Balance" is Crucial: The most striking insight is the necessity of a balanced relationship between kernel decay and bandwidth eigenvalue magnitudes. It's not enough for the bandwidth to shrink; the kernel must also "decay fast enough" to prevent unbounded bias, regardless of density smoothness.
  • Kernel Decay as a Simplifier: The paper argues that sufficient kernel decay can relax the need for strong assumptions on the density function. This challenges the conventional wisdom that complex density assumptions are the only way to fix issues arising from Taylor expansions.
  • Non-Compact Kernels are Fine (with caveats): The work shows that kernels with non-compact support can be handled theoretically, provided their decay properties are adequate. This broadens the applicability of KDE theory.
  • Eigenvalue Discrepancy Matters: The paper highlights that the individual rates at which bandwidth eigenvalues shrink are important, not just their collective behavior (like the determinant or norm). A large discrepancy can lead to unboundedness if the kernel decay isn't sufficient.

Key Results

  • Theorem 1: Establishes a lower bound on the bias, demonstrating that for any desired decay rate and moment properties, one can construct a radial kernel where the density estimation at a point is lower-bounded by a term involving the bandwidth eigenvalues and a power of the input. This explicitly shows the dependency on the balance between kernel decay and bandwidth structure.
  • Theorem 2: Provides a general formula for the bias of a k-th order kernel estimator with bounded moments, under assumptions on the density's k-th derivatives. This formula includes a remainder term that is bounded by the kernel decay and bandwidth properties.
  • Corollary 1: Under specific conditions (kernel decay faster than $||u||^{-d}$ and balanced bandwidth eigenvalues), the remainder term in the bias is shown to be $o(||h||^k)$, indicating improved bias behavior.

Limitations and Future Work

  • Focus on Bias: The paper primarily focuses on the bias of the KDE estimator. While bias is a critical component of the overall error, the variance is also important for a complete picture of KDE accuracy.
  • Theoretical Nature: The results are theoretical and derived through mathematical analysis. While they provide fundamental insights, empirical validation on real-world datasets would further solidify their practical implications.
  • Future Work: The authors suggest that their Taylor formula approach could potentially be weakened further, for instance, by allowing (k-1)-th derivatives to be absolutely continuous rather than strictly differentiable. This hints at avenues for even more relaxed assumptions on the density. The paper also implicitly points towards further research into constructing optimal kernels that balance decay and moment properties for various bandwidth structures.

Summary updated

Abstract

Kernel Density Estimation is a very popular technique of approximating a density function from samples. The accuracy is generally well-understood and depends, roughly speaking, on the kernel decay and local smoothness of the true density. However concrete statements in the literature are often invoked in very specific settings (simplified or overly conservative assumptions) or miss important but subtle points (e.g. it is common to heuristically apply Taylor's expansion globally without referring to compactness). The contribution of this paper is twofold (a) we demonstrate that, when the bandwidth is an arbitrary invertible matrix going to zero, it is necessary to keep a certain balance between the \emph{kernel decay} and \emph{magnitudes of bandwidth eigenvalues}; in fact, without the sufficient decay the estimates may not be even bounded (b) we give a rigorous derivation of bounds with explicit constants for the bias, under possibly minimal assumptions. This connects the kernel decay, bandwidth norm, bandwidth determinant and density smoothness. It has been folklore that the issue with Taylor's formula can be fixed with more complicated assumptions on the density (for example p. 95 of "Kernel Smoothing" by Wand and Jones); we show that this is actually not necessary and can be handled by the kernel decay alone.
BibTeX
@misc{skorski2019kernel,
  author = {Maciej Skorski},
  title = {{Kernel Density Estimation Bias under Minimal Assumptions}},
  year = {2019},
  eprint = {1901.00331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.00331},
}
Figure