Abstract
This paper develops a new direct approach to approximating suprema of general empirical processes by a sequence of suprema of Gaussian processes, without taking the route of approximating whole empirical processes in the sup-norm. We prove an abstract approximation theorem applicable to a wide variety of statistical problems, such as construction of uniform confidence bands for functions. Notably, the bound in the main approximation theorem is nonasymptotic and allows for functions that index the empirical process to be unbounded and have entropy divergent with the sample size. The proof builds on a new coupling inequality for maxima of sums of random vectors, utilizing Stein’s method for normal approximation and new empirical process techniques. We apply this theorem to local and series empirical processes in nonparametric estimation, demonstrating Gaussian approximation under weak regularity conditions for bandwidth and series function counts.
Introduction
In statistical inference, empirical processes—sums of random functions—are fundamental tools for understanding the behavior of estimators. A central challenge is determining the distribution of the maximum value, or supremum, of these processes. This distribution is critical for constructing uniform confidence bands, which provide a range of values that contain the entire true function with a specified level of certainty (e.g., 95%).
Historically, the standard approach to this problem relied on the Komlós-Major-Tusnády (KMT) construction. KMT seeks to approximate the entire path of an empirical process with a Gaussian process. While powerful, KMT requires stringent conditions: the functions must be bounded, and the underlying distributions must be exceptionally well-behaved. In modern nonparametric statistics, such as kernel or series estimation, these conditions are often violated. As we collect more data, the complexity of our models grows, leading to "non-Donsker" function classes where traditional path-wise approximations fail. This paper introduces a direct method to bypass these limitations.
The Direct Approximation Method
The core innovation of this research is the shift from path-wise approximation to a direct focus on the supremum. The authors argue that if the goal is to understand the "highest peak" of a process (the supremum), one does not need to perfectly approximate every point on the map. By focusing specifically on the supremum, the researchers can establish Gaussian approximations under much weaker conditions than those required by KMT.
Technical Pillars
The paper’s theoretical framework rests on several key pillars:
- A New Coupling Inequality: The authors developed a novel method to "couple" the maximum of a sum of random vectors to the maximum of a Gaussian vector. This allows for a direct comparison of their distributions.
- Stein’s Method: To bound the errors in this coupling inequality, the authors utilized Stein’s method for normal approximation. This is a sophisticated probabilistic tool that provides robust error bounds for distributional approximations.
- Non-asymptotic Bounds: Unlike many statistical theorems that only describe behavior as the sample size $n$ approaches infinity, this paper provides explicit bounds for fixed $n$. This makes the results more relevant for practical applications with finite datasets.
Theorem 2.1: The Abstract Approximation Theorem
The "master theorem" of the paper, Theorem 2.1, provides a general framework for bounding the difference between the empirical supremum $Z$ and its Gaussian counterpart $\tilde{Z}$. This theorem is designed to handle modern statistical complexities, including:
- Divergent Entropy: The theory remains valid even as the complexity (entropy) of the function class grows with the sample size.
- Unbounded Noise: Unlike previous landmark results, this method allows for functions that are not strictly bounded, making it applicable to regression models with Gaussian or sub-Gaussian noise.
- Dimension Independence: In specific contexts, the bounds are relatively insensitive to the number of variables, preventing the approximation quality from degrading exponentially as features are added.
Applications in Nonparametric Estimation
The authors demonstrate the utility of their theorem by applying it to two common nonparametric frameworks:
1. Local Empirical Processes
Used in kernel density and kernel regression estimation, these processes are essential for smoothing data. The authors show that for kernel density estimation in $d$ dimensions, their error rates outperform previous state-of-the-art methods (such as those by Rio) when $d \ge 4$. The approximation error typically scales around $(nh_n^d)^{-1/6}$, where $h_n$ is the bandwidth.
2. Series Empirical Processes
In series or sieve estimation (e.g., using polynomials or splines to approximate functions), the complexity is governed by the number of basis functions $K_n$. The paper establishes that the Gaussian approximation holds with an error rate of approximately $(K_n/n)^{-1/6}$, providing a theoretical foundation for constructing confidence bands in high-dimensional series models.
Conclusion
By providing a direct path to Gaussian approximation, this paper expands the toolkit for researchers working with complex, high-dimensional, and non-bounded data. While the results are primarily "in probability" bounds, they offer a robust theoretical basis for the bootstrap methods often used in practice to compute these suprema. This work ensures that even as our models become more complex, our ability to provide rigorous, uniform confidence intervals remains intact.
Summary updated