Introduction
In supervised learning, practitioners constantly navigate the "bias-variance dilemma." Models that are too simple suffer from high bias, failing to capture underlying data patterns. Conversely, overly complex models suffer from high variance, becoming excessively sensitive to the specific training sample. While the machine learning community frequently relies on "ensembling"—combining multiple models—to reduce variance, the theoretical justifications for this practice are often empirical or rely on asymptotic limits assuming infinite data.
This paper bridges the gap in understanding how model combinations behave in the practical reality of finite, noisy datasets. By providing an exact analytical study of the bias-variance tradeoff when combining multiple least squares estimators, the research asks: How do bias and variance evolve as a function of sample size? Furthermore, is it more effective to train all models on the same dataset, or to partition the data among them?
Methodology and Data Setup
The study focuses on linear least squares regression. To derive precise analytical answers, the author employs the thermodynamic limit, a framework borrowed from statistical physics. In this regime, both the number of data points ($n$) and the number of dimensions ($d$) approach infinity, but their ratio ($\alpha = n/d$) remains fixed. This mathematical setup enables the calculation of exact "learning curves" for both bias and variance.
The paper evaluates two primary strategies for ensembling $K$ estimators:
- Fully Overlapping: Every estimator is trained on the exact same dataset.
- Non-Overlapping: The dataset is partitioned into $K$ independent subsets, with each estimator trained on a distinct subset.
Crucially, the analysis considers the unrealizable case—where the true data-generating function may be non-linear, but the model attempting to fit it is linear. This mirrors real-world scenarios where models are inherently simpler than the underlying phenomena. Because the paper is highly theoretical, it relies on a synthetic data generation model rather than standard empirical benchmarks. Inputs ($x$) and target weights ($w_0$) are drawn from multi-dimensional Gaussian distributions, and targets ($y$) are generated by a potentially non-linear function $g(x) + \eta$, where $\eta$ represents Gaussian noise.
Key Insights and Results
The exact calculations yield several counterintuitive insights regarding when and how to ensemble models:
- The "Independence Dividend" for Small Data: The most striking finding is that for small, noisy datasets, splitting data into independent, non-overlapping chunks often outperforms training a single model on all available data. Although each individual model suffers from higher bias (because its effective sample size drops from $\alpha$ to $\alpha/K$), the total variance of the averaged ensemble drops so significantly that the overall expected error decreases.
- The Moore-Penrose Limit: In the underdetermined regime ($n < d$), infinite solutions exist. The paper demonstrates that the Moore-Penrose pseudo-inverse (the solution with the smallest norm) is the optimal single estimator. Combining multiple different solutions trained on the same data cannot outperform this single minimal-norm solution.
- Optimal Ensemble Size ($K^*$): The optimal number of estimators is highly dependent on the noise level ($\sigma^2$). For very small $\alpha$, the optimal number of independent estimators is approximately $K^* \approx (g^2 + \sigma^2) / \bar{u}g^2$. If the data is highly noisy, more estimators are beneficial; if it is clean, a single estimator suffices.
- Ensembling is Not Universally Optimal: The research debunks the heuristic that "more experts are always better." Once the sample size crosses a certain threshold ($\alpha > 1$), a single estimator trained on the entire dataset consistently outperforms any combination of subset-trained estimators. As $\alpha \to \infty$, all uniform mixtures of estimators converge to the same error rate, but their finite-sample behaviors differ drastically.
Limitations and Future Directions
While providing rigorous exact curves for bias and variance, the study has limitations. The most significant is its strict focus on linear estimators. While the findings may extend to simple non-linearities (like a single neuron), analyzing deep, multi-layered neural networks involves significantly more complexity.
Additionally, the paper explores the extremes of data partitioning: fully overlapping versus completely non-overlapping. In practice, techniques like Bagging (Bootstrap Aggregating) utilize partially overlapping subsets, which may offer a middle ground that this paper does not fully explore. Finally, the study primarily investigates uniform mixtures where each estimator receives an equal weight ($1/K$). Determining optimal non-uniform weighting when the underlying data distribution is unknown remains an open challenge for future research.
Summary updated