A Weak Law of Large Numbers for Empirical Measures via Stein's Method

Gesine ReinertThe Annals of Probability 1995 · DOI

This paper, published in The Annals of Probability in 1995, represents a significant step in expanding the toolkit of probability theory. Gesine Reinert demonstrates how Stein’s method—a powerful technique usually reserved for proving Central Limit Theorems—can be elegantly adapted to prove the Weak Law of Large Numbers (WLLN) for empirical measures.

TL;DR

Gesine Reinert extends Stein’s method to prove the Weak Law of Large Numbers for empirical measures, providing a robust framework that handles both independent and weakly dependent data. Unlike classical approaches, this method naturally yields explicit rates of convergence and applies to complex stochastic processes like immigration-death models.

What Problem Does This Paper Address?

The Weak Law of Large Numbers (WLLN) is a cornerstone of statistics, typically stating that the average of a sequence of random variables converges to their mean. When dealing with empirical measures (which describe the distribution of data points in a space rather than just their numerical average), proving this convergence is more complex.

Traditional proofs often rely on Glivenko-Cantelli theorems or characteristic functions, which can be difficult to apply when the data points are not independent. Reinert addresses the need for a more flexible, "local" approach that can:

  1. Handle random elements in general topological spaces (locally compact Hausdorff spaces).
  2. Account for weak dependencies between variables.
  3. Provide rates of convergence (how fast the law "settles down") rather than just proving that it happens eventually.

Methods

The paper utilizes Stein’s Method, specifically the Generator Approach.

  1. The Stein Equation: The core idea is to find an operator $\mathscr{A}$ such that a random variable $W$ has the target distribution if and only if $\mathbb{E}[\mathscr{A}f(W)] = 0$ for a wide class of functions $f$.
  2. The Generator Method: Reinert identifies $\mathscr{A}$ as the generator of a Markov process that has the desired limit measure $\mu$ as its stationary distribution. By "running" this process, one can solve the Stein equation $g(x) - \mathbb{E}[g(X)] = \mathscr{A}f(x)$.
  3. Taylor Expansions: To handle dependencies, the author uses Taylor expansions of the Stein solution. This allows her to bound the error terms by looking at how much a single variable $X_i$ influences the rest of the system.
  4. Zolotarev Metrics: To quantify the distance between the empirical measure and the limit, the paper employs Zolotarev semimetrics, which are particularly well-suited for functional analysis in probability.

Datasets

As a theoretical mathematics paper, no empirical datasets were used. Instead, the author validates the method through mathematical examples:

  • Dissociated Families: A specific type of weakly dependent random array.
  • Immigration-Death Processes: A stochastic model representing population dynamics, used here to illustrate how the method proves convergence for paths of stochastic processes.

Key Insights and Ideas

  • Stein’s Method is Versatile: While usually associated with Normal or Poisson approximations, Reinert shows it works perfectly for "degenerate" limits (where the distribution collapses to a single point, as in the WLLN).
  • Automatic Rates of Convergence: One of the biggest "aha" moments is that the Stein framework "quasi-automatically" provides error bounds. For i.i.d. (independent and identically distributed) cases, the paper shows the expected order of magnitude for the error is $O(1/n)$.
  • A Unified Framework for Dependency: By focusing on the generator of a Markov process, the author provides a recipe for handling dependencies: if you can bound the local interaction between variables, you can prove the global law of large numbers.

Key Results

  • Theorem 2.1: Establishes the WLLN for independent random elements in a general space.
  • Corollary 2.4 & 2.6: Extends the result to dependent variables, requiring only that the "average" interaction remains small.
  • Quantitative Bounds: The paper provides explicit inequalities for the Zolotarev metric. For the i.i.d. case, the distance between the empirical measure and the limit is bounded by $2/n$.
  • Stochastic Application: The method successfully proves the convergence of the path behavior for an immigration-death process, a result that is often much harder to reach using purely frequentist or martingale-based methods.

Limitations and Future Work

  • Weak vs. Strong Laws: The paper focuses on the Weak Law (convergence in probability). While the author mentions Glivenko-Cantelli results (which are "Strong" laws), this specific Stein-based approach does not immediately yield almost sure convergence.
  • Choice of Generator: While the generator method is powerful, the author notes that for a given distribution, there may be many possible Markov processes to choose from. Finding the "optimal" one to get the best convergence rates remains an area requiring "good intuition."
  • Topological Constraints: The results are derived for locally compact Hausdorff spaces with a countable basis. While broad, this excludes some more exotic infinite-dimensional spaces.

Summary updated

Abstract

Let $E$ be a locally compact Hausdorff space with countable basis and let $(X_i)_{\i\in\mathbb{N}}$ be a family of random elements on $E$ with $(1/n) \sum^n_{i=1} \mathscr{L}(X_i) \Rightarrow^v \mu (n \rightarrow \infty)$ for a measure $\mu$ with $\|\mu\| \leq 1$. Conditions are derived under which $\mathscr{L} ((1/n) \sum^n_{i=1} \delta_{Xi}) \Rightarrow^w \delta_\mu(n \rightarrow \infty)$, where $\delta_x$ denotes the Dirac measure at $x$. The proof being based on Stein's method, there are generalisastions that allow for weak dependence between the $X_i$'s. As examples, a dissociated family and an immigration-death process are considered. The latter illustrates the possible applications in proving convergence of stochastic processes.
BibTeX
@article{reinert1995weak,
  author = {Gesine Reinert},
  title = {{A Weak Law of Large Numbers for Empirical Measures via Stein's Method}},
  journal = {The Annals of Probability},
  year = {1995},
  doi = {10.1214/aop/1176988389},
}
Figure