Convergence of Weighted Sums Under Dependence

12 concepts7 papers1 proposal

The classical Law of Large Numbers (LLN) is a cornerstone of statistics: it guarantees that if you take an equally weighted average of independent, identically distributed (i.i.d.) random variables, the sample average will inevitably settle down to the true population mean as you collect more data.

But real-world applications rarely enjoy the luxury of both equal weights and perfect independence. In sequential Monte Carlo simulations, particles are assigned unequal weights to track evolving target distributions. In optimization, algorithms intentionally scale successive steps to discount early, noisy errors. In time-series analysis or spatial modeling, observations are inherently correlated or "mixed" over time and space.

When we replace a simple average with a weighted sum—where each observation $X_i$ is scaled by a specific weight $w_i$—the standard mathematical machinery breaks down. To understand why and how these systems still converge, we must look at the delicate interplay between two forces: the growth rate of the weights and the structural dependence of the random variables.

The Core Tension: Weights vs. Dependence

To understand why weighted convergence is challenging, consider the variance of a weighted sum of random variables:

$$\text{Var}\left(\sum w_i X_i\right) = \sum w_i^2 \text{Var}(X_i) + \sum_{i \neq j} w_i w_j \text{Cov}(X_i, X_j)$$

In the classical i.i.d. setting, the covariance terms on the right are zero, and the weights are uniformly $1/n$, causing the total variance to vanish at a predictable rate of $1/n$.

However, if the weights $w_i$ vary dramatically, a few massive weights can dominate the sum, preventing individual fluctuations from canceling each other out. Worse, if the variables are positively dependent ($\text{Cov}(X_i, X_j) > 0$), they can fluctuate in unison, compounding the total variance and causing the estimator to explode or drift erratically.

To guarantee almost sure convergence—the mathematical promise that an estimator will approach its true value with probability one—the weights must be strictly constrained relative to the underlying dependence structure of the data.

Harnessing Negative Dependence for Stability

Surprisingly, dependence is not always an obstacle. Certain types of non-independent sequences possess structural properties that actively assist convergence. Two prominent examples are Negative Quadrant Dependence (NQD) and Negatively Superadditive Dependence (NSD).

In an NQD or NSD sequence, variables are loosely speaking "negatively correlated": if one variable takes a large value, the others are more likely to take small values. This built-in counterbalancing act means that individual deviations naturally cancel one another out, stabilizing the partial sums even without strict independence.

In Generalization of Almost Sure Convergence Properties of Pairwise NQD Random Sequences, WU Yan-chun et al. established that a sequence of pairwise NQD variables will converge almost surely to its mean if the sum of its variances, scaled by a squared slowly varying function, remains finite. Rather than forcing weights to adhere to rigid power laws, this approach uses slowly varying functions to provide a highly flexible mathematical envelope, proving that the negative dependence is strong enough to keep the sequence's path stable.

Similarly, Zhiyong Chen et al. extended this logic to NSD systems in Strong Convergence for weighed sums of negatively superadditive dependent random variables. The authors mapped out the precise boundaries between weight sequences and moment conditions. They demonstrated that by constraining the growth of the weights relative to the heavy-tailedness (moments) of the distributions, the normalized maximum of the partial sums is guaranteed to vanish almost surely.

Tracking Convergence in Particle Systems

The practical stakes of this theory are vividly apparent in Sequential Monte Carlo (SMC) methods, or particle filters. These algorithms track complex, changing distributions by maintaining a population of "particles," each representing a possible state, modified by an importance weight.

As the number of particles approaches infinity, do these weighted averages actually converge to the true target distribution? In Limit theorems for weighted samples with applications to sequential Monte Carlo methods, Randal Douc et al. provided the foundational proofs. The authors established both a Law of Large Numbers and a Central Limit Theorem (CLT) for weighted particle systems, proving that the scaled difference between the weighted particle estimate and the true expectation converges to a normal distribution. This provides the theoretical justification required to trust particle filters in high-stakes tracking and inference tasks.

[True Target Distribution] 
       ▲
       │  (Converges via LLN / CLT as particles → ∞)
  [Weighted Sum of Particles]  ◄─── Balance ───► [Weight Fluctuations]
       │                                               │
       ▼                                               ▼
(Stable if weights are bounded)               (Collapses if variance 
                                               grows exponentially)

However, this convergence is highly sensitive to how the weights behave. If the variance of the weights grows unchecked, a single particle might end up carrying a weight of 0.999, rendering the rest of the sample useless—a phenomenon known as weight collapse.

In The inefficiency of re-weighted sampling and the curse of system size in high-order path integration, Guy A. R. Brain et al. demonstrated a severe boundary condition of this paradigm. By analyzing the joint Gaussian distribution of observables and log-weights, the authors showed that in high-order path integrals, statistical uncertainty scales exponentially with system size. This "curse of system size" occurs because the fluctuations of the weights overwhelm the estimator, illustrating that theoretical convergence at infinity does not guarantee practical utility in large-scale physical simulations.

Proving convergence becomes even more intricate when the weights are not assigned independently but instead depend on the positions of all other points in the system. For instance, Guillaume Gautier et al. explored this in On two ways to use determinantal point processes for Monte Carlo integration. When evaluating the Ermakov-Zolotukhin estimator, the quadrature weights are globally coupled to all nodes. Because the weights cannot be decoupled, standard analytical approaches fail, making the derivation of a Central Limit Theorem an ongoing theoretical hurdle.

Designing Weights for Algorithmic Progress

While statisticians often study weights as an inherent property of collected data, optimization researchers use weights as a deliberate design tool to force convergence in unstable algorithms.

Consider saddle-point problems, which underpin modern machine learning frameworks like Generative Adversarial Networks (GANs). Standard gradient steps frequently oscillate or diverge in these environments. To stabilize them, practitioners often average the algorithm's iterates over time.

In Increasing Iterate Averaging for Solving Saddle-Point Problems, Christian Kroer et al. introduced a weighted telescoping sum analysis. Instead of treating every iterate equally, they applied strictly non-decreasing weights to successive steps.

$$\text{Weighted Average} = \frac{\sum w_i \cdot \text{Iterate}_i}{\sum w_i}$$

By systematically increasing the weights over time, the influence of early, highly inaccurate iterates is minimized. The mathematical framework proves that this deliberate weighting scheme preserves the overall convergence rate while smoothing out the chaotic oscillations typical of saddle-point dynamics.

Finally, these theoretical tools serve as a reusable foundation for proving the reliability of practical statistical models where data points are temporally or spatially linked. In Strong convergence for weighted sums of ($\alpha$, $\beta$)-mixing random variables and application to simple linear EV regression model, Wenjing Hu et al. analyzed sequences where variables are $(\alpha, \beta)$-mixing—meaning that observations collected far apart in time become progressively closer to independent.

By establishing strong laws for the weighted sums of these mixing variables, the authors were able to prove the consistency of estimators in linear errors-in-variables (EV) regression models. Because many complex estimators are fundamentally structured as weighted sums, these abstract convergence laws guarantee that as your sample size grows, your parameter estimates will almost surely track the true physical state of the system.

Key Foundational Work

Generalization of Almost Sure Convergence Properties of Pairwise NQD Random Sequences (2011) — contributed 3 concept elements to this direction.
Limit theorems for weighted samples with applications to sequential Monte Carlo methods (2008) — contributed 3 concept elements to this direction.
The inefficiency of re-weighted sampling and the curse of system size in high-order path integration (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_weighted_sum_convergence_68b46522,
  title = {weighted sum convergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weighted-sum-convergence/rdr_68b46522359648cda8fb33b72319e7a0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_weighted_sum_convergence_68b46522_v1,
  title = {weighted sum convergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weighted-sum-convergence/rdr_68b46522359648cda8fb33b72319e7a0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake