Gaussian approximation of suprema of empirical processes

Victor Chernozhukov, Denis Chetverikov, Kengo KatoThe Annals of Statistics 2014 · arXiv

Abstract

This paper develops a new direct approach to approximating suprema of general empirical processes by a sequence of suprema of Gaussian processes, without taking the route of approximating whole empirical processes in the sup-norm. We prove an abstract approximation theorem applicable to a wide variety of statistical problems, such as construction of uniform confidence bands for functions. Notably, the bound in the main approximation theorem is nonasymptotic and allows for functions that index the empirical process to be unbounded and have entropy divergent with the sample size. The proof builds on a new coupling inequality for maxima of sums of random vectors, utilizing Stein’s method for normal approximation and new empirical process techniques. We apply this theorem to local and series empirical processes in nonparametric estimation, demonstrating Gaussian approximation under weak regularity conditions for bandwidth and series function counts.

Introduction

In statistical inference, empirical processes—sums of random functions—are fundamental tools for understanding the behavior of estimators. A central challenge is determining the distribution of the maximum value, or supremum, of these processes. This distribution is critical for constructing uniform confidence bands, which provide a range of values that contain the entire true function with a specified level of certainty (e.g., 95%).

Historically, the standard approach to this problem relied on the Komlós-Major-Tusnády (KMT) construction. KMT seeks to approximate the entire path of an empirical process with a Gaussian process. While powerful, KMT requires stringent conditions: the functions must be bounded, and the underlying distributions must be exceptionally well-behaved. In modern nonparametric statistics, such as kernel or series estimation, these conditions are often violated. As we collect more data, the complexity of our models grows, leading to "non-Donsker" function classes where traditional path-wise approximations fail. This paper introduces a direct method to bypass these limitations.

The Direct Approximation Method

The core innovation of this research is the shift from path-wise approximation to a direct focus on the supremum. The authors argue that if the goal is to understand the "highest peak" of a process (the supremum), one does not need to perfectly approximate every point on the map. By focusing specifically on the supremum, the researchers can establish Gaussian approximations under much weaker conditions than those required by KMT.

Technical Pillars

The paper’s theoretical framework rests on several key pillars:

  1. A New Coupling Inequality: The authors developed a novel method to "couple" the maximum of a sum of random vectors to the maximum of a Gaussian vector. This allows for a direct comparison of their distributions.
  2. Stein’s Method: To bound the errors in this coupling inequality, the authors utilized Stein’s method for normal approximation. This is a sophisticated probabilistic tool that provides robust error bounds for distributional approximations.
  3. Non-asymptotic Bounds: Unlike many statistical theorems that only describe behavior as the sample size $n$ approaches infinity, this paper provides explicit bounds for fixed $n$. This makes the results more relevant for practical applications with finite datasets.

Theorem 2.1: The Abstract Approximation Theorem

The "master theorem" of the paper, Theorem 2.1, provides a general framework for bounding the difference between the empirical supremum $Z$ and its Gaussian counterpart $\tilde{Z}$. This theorem is designed to handle modern statistical complexities, including:

  • Divergent Entropy: The theory remains valid even as the complexity (entropy) of the function class grows with the sample size.
  • Unbounded Noise: Unlike previous landmark results, this method allows for functions that are not strictly bounded, making it applicable to regression models with Gaussian or sub-Gaussian noise.
  • Dimension Independence: In specific contexts, the bounds are relatively insensitive to the number of variables, preventing the approximation quality from degrading exponentially as features are added.

Applications in Nonparametric Estimation

The authors demonstrate the utility of their theorem by applying it to two common nonparametric frameworks:

1. Local Empirical Processes

Used in kernel density and kernel regression estimation, these processes are essential for smoothing data. The authors show that for kernel density estimation in $d$ dimensions, their error rates outperform previous state-of-the-art methods (such as those by Rio) when $d \ge 4$. The approximation error typically scales around $(nh_n^d)^{-1/6}$, where $h_n$ is the bandwidth.

2. Series Empirical Processes

In series or sieve estimation (e.g., using polynomials or splines to approximate functions), the complexity is governed by the number of basis functions $K_n$. The paper establishes that the Gaussian approximation holds with an error rate of approximately $(K_n/n)^{-1/6}$, providing a theoretical foundation for constructing confidence bands in high-dimensional series models.

Conclusion

By providing a direct path to Gaussian approximation, this paper expands the toolkit for researchers working with complex, high-dimensional, and non-bounded data. While the results are primarily "in probability" bounds, they offer a robust theoretical basis for the bootstrap methods often used in practice to compute these suprema. This work ensures that even as our models become more complex, our ability to provide rigorous, uniform confidence intervals remains intact.

Summary updated

Abstract

This paper develops a new direct approach to approximating suprema of general empirical processes by a sequence of suprema of Gaussian processes, without taking the route of approximating whole empirical processes in the sup-norm. We prove an abstract approximation theorem applicable to a wide variety of statistical problems, such as construction of uniform confidence bands for functions. Notably, the bound in the main approximation theorem is nonasymptotic and the theorem does not require uniform boundedness of the class of functions. The proof of the approximation theorem builds on a new coupling inequality for maxima of sums of random vectors, the proof of which depends on an e.ective use of Stein's method for normal approximation, and some new empirical process techniques. We study applications of this approximation theorem to local and series empirical processes arising in nonparametric estimation via kernel and series methods, where the classes of functions change with the sample size and are non-Donsker. Importantly, our new technique is able to prove the Gaussian approximation for the supremum type statistics under weak regularity conditions, especially concerning the bandwidth and the number of series functions, in those examples.
BibTeX
@article{chernozhukov2014gaussian,
  author = {Victor Chernozhukov and Denis Chetverikov and Kengo Kato},
  title = {{Gaussian approximation of suprema of empirical processes}},
  journal = {The Annals of Statistics},
  year = {2014},
}
Figure