Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective

Ye Su, Mingrui Ye, Yining Wang, Jipeng Guo, Yong LiuarXiv 2026 · DOI

TL;DR

This paper introduces an exact, finite-sample method to decompose the variance of subagging (a type of ensemble learning) by viewing it as a spectral filter. This reveals that subagging doesn't reduce variance uniformly but rather attenuates higher-order "noise" components, a crucial insight for understanding and optimizing ensemble methods, especially with modern high-capacity models.

What Problem Does This Paper Address?

Ensemble methods like bagging and subagging are workhorses in machine learning for reducing prediction variance. However, the exact way these methods interact with the intrinsic complexity of a base learner, particularly in finite samples, has lacked a precise mathematical characterization. Standard resampling ratios (like 0.632) are often used as defaults, but it's unclear why they work or how they should be adapted for different models. Modern high-capacity models, like deep neural networks or unpruned decision trees, can "interpolate" data, meaning their variance is heavily concentrated in complex, high-order interactions. Existing theoretical approaches often rely on asymptotic limits, which can obscure these critical finite-sample behaviors. This paper aims to bridge this gap by providing an exact, finite-sample decomposition of subagging's variance reduction mechanism.

Methods

The core of this paper is the development of an exact, finite-sample variance decomposition for subagging. The authors leverage the Hoeffding-ANOVA decomposition to break down the variance of a base learner into components corresponding to different orders of interaction among training samples. They then show that subagging acts as a deterministic low-pass spectral filter. This means it preserves low-order structural signals while attenuating higher-order interaction variance by a geometric factor related to the subsampling ratio ($\alpha$).

Key methodological contributions include:

  • Exact Finite-Sample Variance Identity: A theorem providing a precise formula for the variance of subagging that holds for any symmetric base learner, regardless of sample size or smoothness assumptions.
  • The Spectral Filtering Law: An analytical characterization of how subagging attenuates variance at each interaction order, showing a geometric decay governed by $\alpha^c$.
  • Complexity-Guided Adaptive Subsampling (CGAS): An algorithm that operationalizes these theoretical insights by dynamically calibrating the subsampling ratio based on a proxy for the base learner's structural complexity (e.g., tree depth). This avoids the need for full spectral estimation.

Datasets

The paper evaluates its framework on a diverse suite of nine public datasets sourced from OpenML:

  • yprop_4_1
  • wine_quality
  • superconduct
  • dataset_sale
  • coil_20
  • elevators
  • california_housing
  • sarcos
  • 2dplanes

These datasets vary in sample size and feature dimensions, allowing for robust empirical validation across different data scales and complexity regimes.

Key Insights and Ideas

  • Subagging as a Spectral Filter: The most significant insight is that subagging doesn't uniformly reduce variance. Instead, it acts like a low-pass filter on the base learner's "variance spectrum," attenuating high-frequency (high-order interaction) noise while preserving low-frequency (low-order structural) signals.
  • Optimal Ratio Depends on Complexity: The optimal subsampling ratio ($\alpha^*$) is not fixed but is monotonically non-increasing with the base learner's interaction complexity. Simple models need larger $\alpha$ to avoid bias, while complex interpolators require smaller $\alpha$ to suppress high-order noise.
  • Default Ratios are Suboptimal for Interpolators: The common default subsampling ratios (e.g., $\alpha \approx 0.632$) are often too large for high-capacity interpolators, leading to under-regularization and poor generalization.
  • Geometric Attenuation of Noise: The attenuation factor for higher-order interactions decays geometrically with $\alpha^c$, meaning even moderate values of $\alpha$ can significantly suppress noise in complex models.

Key Results

  • Theoretical Variance Decomposition: The paper derives an exact formula for subagging's variance reduction, showing it scales with $\gamma_c(n,k) \cdot \zeta_c(x)$, where $\gamma_c(n,k)$ is the attenuation factor and $\zeta_c(x)$ is the variance component at order $c$.
  • Spectral Filtering Law: The attenuation factor $\gamma_c(n,k)$ is shown to be bounded by $\alpha^c$, converging to $\alpha^c$ in the large-sample limit.
  • Optimal Ratio Shift: Experiments with CGAS demonstrate a clear shift in the optimal subsampling ratio ($\alpha^*$) across complexity regimes. For low-complexity models (shallow trees), $\alpha^*$ clusters near the classical 0.632. For high-complexity models (deep trees), $\alpha^*$ shifts significantly lower (0.1-0.3).
  • Performance Gains: CGAS consistently outperforms fixed-rate subagging baselines (0.632 and 0.8) in the high-complexity regime, achieving substantial reductions in MSE and MAE (e.g., up to 20.7% MSE reduction on the 2dplanes dataset). This holds across different base learners like KNN.
  • Bootstrap Extension: Applying the CGAS-calibrated ratio to bootstrap sampling (Random Forests) also shows significant improvements in high-complexity regimes, validating the general applicability of the spectral filtering concept.

Limitations and Future Work

  • Subagging vs. Bagging: The exact variance decomposition is derived for subagging (sampling without replacement) to maintain multilinear orthogonality. Extending this to bagging (sampling with replacement) is a non-trivial challenge due to complex structural entanglements in Hoeffding projections caused by duplicate samples.
  • Computational Complexity: While CGAS avoids full spectral estimation, estimating the entire spectrum for theoretical calibration remains computationally prohibitive.
  • Future Work: The authors suggest developing generalized orthogonal bases to disentangle correlation structures in bagging, aiming for a complete finite-sample characterization of sampling with replacement. They also aim to explore how this spectral filtering framework can inform the design of new ensemble methods.

Summary updated

BibTeX
@misc{su2026exact,
  author = {Ye Su and Mingrui Ye and Yining Wang and Jipeng Guo and Yong Liu},
  title = {{Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective}},
  year = {2026},
  eprint = {2604.10469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.10469},
}
Figure